* [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
@ 2022-05-25 9:11 Joel Hutton
2022-05-27 13:23 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Joel Hutton @ 2022-05-25 9:11 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Richard Biener, gcc-patches
Ping!
Just checking there is still interest in this. I'm assuming you've been busy with release.
Joel
> -----Original Message-----
> From: Joel Hutton
> Sent: 13 April 2022 16:53
> To: Richard Sandiford <richard.sandiford@arm.com>
> Cc: Richard Biener <rguenther@suse.de>; gcc-patches@gcc.gnu.org
> Subject: [vect-patterns] Refactor widen_plus/widen_minus as internal_fns
>
> Hi all,
>
> These patches refactor the widening patterns in vect-patterns to use
> internal_fn instead of tree_codes.
>
> Sorry about the delay, some changes to master made it a bit messier.
>
> Bootstrapped and regression tested on aarch64.
>
> Joel
>
> > > diff --git a/gcc/tree-vect-patterns.c b/gcc/tree-vect-patterns.c
> > > index 854cbcff390..4a8ea67e62f 100644
> > > --- a/gcc/tree-vect-patterns.c
> > > +++ b/gcc/tree-vect-patterns.c
> > > @@ -1245,7 +1245,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
> > > static gimple * vect_recog_widen_op_pattern (vec_info *vinfo,
> > > stmt_vec_info last_stmt_info, tree *type_out,
> > > - tree_code orig_code, tree_code wide_code,
> > > + tree_code orig_code, code_helper
> > wide_code_or_ifn,
> >
> > I think it'd be better to keep the original “wide_code” name and try
> > to remove as many places as possible in which switching based on
> > tree_code or internal_fn is necessary. The recent gimple-match.h
> > patches should help with that, but more routines might be needed.
>
> Done.
>
> > > @@ -1309,8 +1310,16 @@ vect_recog_widen_op_pattern (vec_info
> *vinfo,
> > > 2, oprnd, half_type, unprom, vectype);
> > >
> > > tree var = vect_recog_temp_ssa_var (itype, NULL);
> > > - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
> > > - oprnd[0], oprnd[1]);
> > > + gimple *pattern_stmt;
> > > + if (wide_code_or_ifn.is_tree_code ())
> > > + pattern_stmt = gimple_build_assign (var, wide_code_or_ifn,
> > > + oprnd[0], oprnd[1]);
> > > + else
> > > + {
> > > + internal_fn fn = as_internal_fn ((combined_fn) wide_code_or_ifn);
> > > + pattern_stmt = gimple_build_call_internal (fn, 2, oprnd[0], oprnd[1]);
> > > + gimple_call_set_lhs (pattern_stmt, var);
> > > + }
> >
> > For example, I think we should hide this inside a new:
> >
> > gimple_build (var, wide_code, oprnd[0], oprnd[1]);
> >
> > that works directly on code_helper, similarly to the new code_helper
> > gimple_build interfaces.
>
> Done.
>
> > > @@ -4513,14 +4513,16 @@ vect_gen_widened_results_half (vec_info
> > *vinfo, enum tree_code code,
> > > tree new_temp;
> > >
> > > /* Generate half of the widened result: */
> > > - gcc_assert (op_type == TREE_CODE_LENGTH (code));
> > > if (op_type != binary_op)
> > > vec_oprnd1 = NULL;
> > > - new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0,
> > vec_oprnd1);
> > > + if (ch.is_tree_code ())
> > > + new_stmt = gimple_build_assign (vec_dest, ch, vec_oprnd0,
> > vec_oprnd1);
> > > + else
> > > + new_stmt = gimple_build_call_internal (as_internal_fn
> > > + ((combined_fn)
> > ch),
> > > + 2, vec_oprnd0, vec_oprnd1);
> >
> > Similarly here. I guess the combined_fn/internal_fn path will also
> > need to cope with null trailing operands, for consistency with the tree_code
> one.
> >
>
> Done.
>
> > > @@ -4744,31 +4747,28 @@ vectorizable_conversion (vec_info *vinfo,
> > > && ! vec_stmt)
> > > return false;
> > >
> > > - gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
> > > - if (!stmt)
> > > + gimple* stmt = stmt_info->stmt;
> > > + if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
> > > return false;
> > >
> > > - if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
> > > - return false;
> > > + if (is_gimple_assign (stmt))
> > > + {
> > > + code_or_ifn = gimple_assign_rhs_code (stmt); } else
> > > + code_or_ifn = gimple_call_combined_fn (stmt);
> >
> > It might be possible to use gimple_extract_op here (only recently added).
> > This would also provide the number of operands directly, instead of
> > needing “op_type”. It would also provide an array of operands.
> >
>
> Done.
>
> > > - code = gimple_assign_rhs_code (stmt);
> > > - if (!CONVERT_EXPR_CODE_P (code)
> > > - && code != FIX_TRUNC_EXPR
> > > - && code != FLOAT_EXPR
> > > - && code != WIDEN_PLUS_EXPR
> > > - && code != WIDEN_MINUS_EXPR
> > > - && code != WIDEN_MULT_EXPR
> > > - && code != WIDEN_LSHIFT_EXPR)
> >
> > Is it safe to drop this check independently of parts 2 and 3?
> > (Genuine question, haven't checked in detail.)
>
> It requires the parts 2 and 3. I've moved that change into this first patch.
>
> > > @@ -4784,7 +4784,8 @@ vectorizable_conversion (vec_info *vinfo,
> > > }
> > >
> > > rhs_type = TREE_TYPE (op0);
> > > - if ((code != FIX_TRUNC_EXPR && code != FLOAT_EXPR)
> > > + if ((code_or_ifn.is_tree_code () && code_or_ifn != FIX_TRUNC_EXPR
> > > + && code_or_ifn != FLOAT_EXPR)
> >
> > I don't think we want the is_tree_code condition here. The existing
> > != should work.
> >
>
> Done.
>
> > > @@ -11856,13 +11888,13 @@ supportable_widening_operation
> (vec_info
> > *vinfo,
> > > if (BYTES_BIG_ENDIAN && c1 != VEC_WIDEN_MULT_EVEN_EXPR)
> > > std::swap (c1, c2);
> > >
> > > - if (code == FIX_TRUNC_EXPR)
> > > + if (code_or_ifn == FIX_TRUNC_EXPR)
> > > {
> > > /* The signedness is determined from output operand. */
> > > optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
> > > optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
> > > }
> > > - else if (CONVERT_EXPR_CODE_P (code)
> > > + else if (CONVERT_EXPR_CODE_P ((tree_code) code_or_ifn)
> >
> > I think this should be as_tree_code (), so that it's safe for internal
> > functions if (tree_code) ever becomes a checked convrsion in future.
> > Same for other instances.
> >
>
> Done.
>
> > > && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
> > > && VECTOR_BOOLEAN_TYPE_P (vectype)
> > > && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype) […] @@
> > > -12000,7 +12031,7 @@ supportable_widening_operation (vec_info
> > *vinfo,
> > > bool
> > > supportable_narrowing_operation (enum tree_code code,
> > > tree vectype_out, tree vectype_in,
> > > - enum tree_code *code1, int *multi_step_cvt,
> > > + void* _code1, int *multi_step_cvt,
> >
> > This might be rehashing an old conversation, sorry, but why does this
> > need to be void?
> >
>
> Reworked to avoid using void*.
>
> > > vec<tree> *interm_types) {
> > > machine_mode vec_mode;
> > > @@ -12013,6 +12044,7 @@ supportable_narrowing_operation (enum
> > tree_code code,
> > > machine_mode intermediate_mode, prev_mode;
> > > int i;
> > > bool uns;
> > > + tree_code * code1 = (tree_code*) _code1;
> > >
> > > *multi_step_cvt = 0;
> > > switch (code)
> > > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index
> > > bd6f334d15f..70c06264c11 100644
> > > --- a/gcc/tree-vectorizer.h
> > > +++ b/gcc/tree-vectorizer.h
> > > @@ -2030,13 +2030,16 @@ extern bool vect_is_simple_use (vec_info *,
> > stmt_vec_info, slp_tree,
> > > enum vect_def_type *,
> > > tree *, stmt_vec_info * = NULL); extern bool
> > > vect_maybe_update_slp_op_vectype (slp_tree, tree); -extern bool
> > > supportable_widening_operation (vec_info *,
> > > - enum tree_code, stmt_vec_info,
> > > - tree, tree, enum tree_code *,
> > > - enum tree_code *, int *,
> > > - vec<tree> *);
> > > +extern bool supportable_widening_operation (vec_info *vinfo,
> > > + code_helper code_or_ifn,
> > > + stmt_vec_info stmt_info,
> > > + tree vectype_out, tree vectype_in,
> > > + code_helper *code_or_ifn1,
> > > + code_helper *code_or_ifn2,
> > > + int *multi_step_cvt,
> > > + vec<tree> *interm_types);
> >
> > Normal style is to keep the variable names out of the header.
> > The documentation lives in the .c file, so in practice, anyone who
> > wants to add a new caller will need to look there anyway.
> >
> > Thanks,
> > Richard
> >
> > > extern bool supportable_narrowing_operation (enum tree_code, tree,
> > tree,
> > > - enum tree_code *, int *,
> > > + void *, int *,
> > > vec<tree> *);
> > >
> > > extern unsigned record_stmt_cost (stmt_vector_for_cost *, int, diff
> > > --git a/gcc/tree.h b/gcc/tree.h index f62c00bc870..346565f84ce
> > > 100644
> > > --- a/gcc/tree.h
> > > +++ b/gcc/tree.h
> > > @@ -6546,5 +6546,31 @@ extern unsigned fndecl_dealloc_argno (tree);
> > > if nonnull, set the second argument to the referenced enclosing
> > > object or pointer. Otherwise return null. */ extern tree
> > > get_attr_nonstring_decl (tree, tree * = NULL);
> > > +/* Helper to transparently allow tree codes and builtin function codes
> > > + exist in one storage entity. */ class code_helper {
> > > +public:
> > > + code_helper () {}
> > > + code_helper (tree_code code) : rep ((int) code) {}
> > > + code_helper (combined_fn fn) : rep (-(int) fn) {}
> > > + operator tree_code () const { return is_tree_code () ?
> > > + (tree_code) rep :
> > > + ERROR_MARK; }
> > > + operator combined_fn () const { return is_fn_code () ?
> > > + (combined_fn) -rep:
> > > + CFN_LAST; }
> > > + bool is_tree_code () const { return rep > 0; }
> > > + bool is_fn_code () const { return rep < 0; }
> > > + int get_rep () const { return rep; }
> > > +
> > > + enum tree_code as_tree_code () const { return is_tree_code () ?
> > > + (tree_code)* this : MAX_TREE_CODES; } combined_fn as_fn_code
> > > + () const { return is_fn_code () ? (combined_fn)
> > *this
> > > + : CFN_LAST;}
> > > +
> > > +private:
> > > + int rep;
> > > +};
> > >
> > > #endif /* GCC_TREE_H */
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-05-25 9:11 [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns Joel Hutton
@ 2022-05-27 13:23 ` Richard Biener
2022-05-31 10:07 ` Joel Hutton
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2022-05-27 13:23 UTC (permalink / raw)
To: Joel Hutton; +Cc: Richard Sandiford, gcc-patches
On Wed, 25 May 2022, Joel Hutton wrote:
> Ping!
>
> Just checking there is still interest in this. I'm assuming you've been
> busy with release.
Can you post an updated patch (after the .cc renaming, and code_helper
now already moved to tree.h).
Thanks,
Richard.
> Joel
>
> > -----Original Message-----
> > From: Joel Hutton
> > Sent: 13 April 2022 16:53
> > To: Richard Sandiford <richard.sandiford@arm.com>
> > Cc: Richard Biener <rguenther@suse.de>; gcc-patches@gcc.gnu.org
> > Subject: [vect-patterns] Refactor widen_plus/widen_minus as internal_fns
> >
> > Hi all,
> >
> > These patches refactor the widening patterns in vect-patterns to use
> > internal_fn instead of tree_codes.
> >
> > Sorry about the delay, some changes to master made it a bit messier.
> >
> > Bootstrapped and regression tested on aarch64.
> >
> > Joel
> >
> > > > diff --git a/gcc/tree-vect-patterns.c b/gcc/tree-vect-patterns.c
> > > > index 854cbcff390..4a8ea67e62f 100644
> > > > --- a/gcc/tree-vect-patterns.c
> > > > +++ b/gcc/tree-vect-patterns.c
> > > > @@ -1245,7 +1245,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
> > > > static gimple * vect_recog_widen_op_pattern (vec_info *vinfo,
> > > > stmt_vec_info last_stmt_info, tree *type_out,
> > > > - tree_code orig_code, tree_code wide_code,
> > > > + tree_code orig_code, code_helper
> > > wide_code_or_ifn,
> > >
> > > I think it'd be better to keep the original ?wide_code? name and try
> > > to remove as many places as possible in which switching based on
> > > tree_code or internal_fn is necessary. The recent gimple-match.h
> > > patches should help with that, but more routines might be needed.
> >
> > Done.
> >
> > > > @@ -1309,8 +1310,16 @@ vect_recog_widen_op_pattern (vec_info
> > *vinfo,
> > > > 2, oprnd, half_type, unprom, vectype);
> > > >
> > > > tree var = vect_recog_temp_ssa_var (itype, NULL);
> > > > - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
> > > > - oprnd[0], oprnd[1]);
> > > > + gimple *pattern_stmt;
> > > > + if (wide_code_or_ifn.is_tree_code ())
> > > > + pattern_stmt = gimple_build_assign (var, wide_code_or_ifn,
> > > > + oprnd[0], oprnd[1]);
> > > > + else
> > > > + {
> > > > + internal_fn fn = as_internal_fn ((combined_fn) wide_code_or_ifn);
> > > > + pattern_stmt = gimple_build_call_internal (fn, 2, oprnd[0], oprnd[1]);
> > > > + gimple_call_set_lhs (pattern_stmt, var);
> > > > + }
> > >
> > > For example, I think we should hide this inside a new:
> > >
> > > gimple_build (var, wide_code, oprnd[0], oprnd[1]);
> > >
> > > that works directly on code_helper, similarly to the new code_helper
> > > gimple_build interfaces.
> >
> > Done.
> >
> > > > @@ -4513,14 +4513,16 @@ vect_gen_widened_results_half (vec_info
> > > *vinfo, enum tree_code code,
> > > > tree new_temp;
> > > >
> > > > /* Generate half of the widened result: */
> > > > - gcc_assert (op_type == TREE_CODE_LENGTH (code));
> > > > if (op_type != binary_op)
> > > > vec_oprnd1 = NULL;
> > > > - new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0,
> > > vec_oprnd1);
> > > > + if (ch.is_tree_code ())
> > > > + new_stmt = gimple_build_assign (vec_dest, ch, vec_oprnd0,
> > > vec_oprnd1);
> > > > + else
> > > > + new_stmt = gimple_build_call_internal (as_internal_fn
> > > > + ((combined_fn)
> > > ch),
> > > > + 2, vec_oprnd0, vec_oprnd1);
> > >
> > > Similarly here. I guess the combined_fn/internal_fn path will also
> > > need to cope with null trailing operands, for consistency with the tree_code
> > one.
> > >
> >
> > Done.
> >
> > > > @@ -4744,31 +4747,28 @@ vectorizable_conversion (vec_info *vinfo,
> > > > && ! vec_stmt)
> > > > return false;
> > > >
> > > > - gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
> > > > - if (!stmt)
> > > > + gimple* stmt = stmt_info->stmt;
> > > > + if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
> > > > return false;
> > > >
> > > > - if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
> > > > - return false;
> > > > + if (is_gimple_assign (stmt))
> > > > + {
> > > > + code_or_ifn = gimple_assign_rhs_code (stmt); } else
> > > > + code_or_ifn = gimple_call_combined_fn (stmt);
> > >
> > > It might be possible to use gimple_extract_op here (only recently added).
> > > This would also provide the number of operands directly, instead of
> > > needing ?op_type?. It would also provide an array of operands.
> > >
> >
> > Done.
> >
> > > > - code = gimple_assign_rhs_code (stmt);
> > > > - if (!CONVERT_EXPR_CODE_P (code)
> > > > - && code != FIX_TRUNC_EXPR
> > > > - && code != FLOAT_EXPR
> > > > - && code != WIDEN_PLUS_EXPR
> > > > - && code != WIDEN_MINUS_EXPR
> > > > - && code != WIDEN_MULT_EXPR
> > > > - && code != WIDEN_LSHIFT_EXPR)
> > >
> > > Is it safe to drop this check independently of parts 2 and 3?
> > > (Genuine question, haven't checked in detail.)
> >
> > It requires the parts 2 and 3. I've moved that change into this first patch.
> >
> > > > @@ -4784,7 +4784,8 @@ vectorizable_conversion (vec_info *vinfo,
> > > > }
> > > >
> > > > rhs_type = TREE_TYPE (op0);
> > > > - if ((code != FIX_TRUNC_EXPR && code != FLOAT_EXPR)
> > > > + if ((code_or_ifn.is_tree_code () && code_or_ifn != FIX_TRUNC_EXPR
> > > > + && code_or_ifn != FLOAT_EXPR)
> > >
> > > I don't think we want the is_tree_code condition here. The existing
> > > != should work.
> > >
> >
> > Done.
> >
> > > > @@ -11856,13 +11888,13 @@ supportable_widening_operation
> > (vec_info
> > > *vinfo,
> > > > if (BYTES_BIG_ENDIAN && c1 != VEC_WIDEN_MULT_EVEN_EXPR)
> > > > std::swap (c1, c2);
> > > >
> > > > - if (code == FIX_TRUNC_EXPR)
> > > > + if (code_or_ifn == FIX_TRUNC_EXPR)
> > > > {
> > > > /* The signedness is determined from output operand. */
> > > > optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
> > > > optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
> > > > }
> > > > - else if (CONVERT_EXPR_CODE_P (code)
> > > > + else if (CONVERT_EXPR_CODE_P ((tree_code) code_or_ifn)
> > >
> > > I think this should be as_tree_code (), so that it's safe for internal
> > > functions if (tree_code) ever becomes a checked convrsion in future.
> > > Same for other instances.
> > >
> >
> > Done.
> >
> > > > && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
> > > > && VECTOR_BOOLEAN_TYPE_P (vectype)
> > > > && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype) [?] @@
> > > > -12000,7 +12031,7 @@ supportable_widening_operation (vec_info
> > > *vinfo,
> > > > bool
> > > > supportable_narrowing_operation (enum tree_code code,
> > > > tree vectype_out, tree vectype_in,
> > > > - enum tree_code *code1, int *multi_step_cvt,
> > > > + void* _code1, int *multi_step_cvt,
> > >
> > > This might be rehashing an old conversation, sorry, but why does this
> > > need to be void?
> > >
> >
> > Reworked to avoid using void*.
> >
> > > > vec<tree> *interm_types) {
> > > > machine_mode vec_mode;
> > > > @@ -12013,6 +12044,7 @@ supportable_narrowing_operation (enum
> > > tree_code code,
> > > > machine_mode intermediate_mode, prev_mode;
> > > > int i;
> > > > bool uns;
> > > > + tree_code * code1 = (tree_code*) _code1;
> > > >
> > > > *multi_step_cvt = 0;
> > > > switch (code)
> > > > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index
> > > > bd6f334d15f..70c06264c11 100644
> > > > --- a/gcc/tree-vectorizer.h
> > > > +++ b/gcc/tree-vectorizer.h
> > > > @@ -2030,13 +2030,16 @@ extern bool vect_is_simple_use (vec_info *,
> > > stmt_vec_info, slp_tree,
> > > > enum vect_def_type *,
> > > > tree *, stmt_vec_info * = NULL); extern bool
> > > > vect_maybe_update_slp_op_vectype (slp_tree, tree); -extern bool
> > > > supportable_widening_operation (vec_info *,
> > > > - enum tree_code, stmt_vec_info,
> > > > - tree, tree, enum tree_code *,
> > > > - enum tree_code *, int *,
> > > > - vec<tree> *);
> > > > +extern bool supportable_widening_operation (vec_info *vinfo,
> > > > + code_helper code_or_ifn,
> > > > + stmt_vec_info stmt_info,
> > > > + tree vectype_out, tree vectype_in,
> > > > + code_helper *code_or_ifn1,
> > > > + code_helper *code_or_ifn2,
> > > > + int *multi_step_cvt,
> > > > + vec<tree> *interm_types);
> > >
> > > Normal style is to keep the variable names out of the header.
> > > The documentation lives in the .c file, so in practice, anyone who
> > > wants to add a new caller will need to look there anyway.
> > >
> > > Thanks,
> > > Richard
> > >
> > > > extern bool supportable_narrowing_operation (enum tree_code, tree,
> > > tree,
> > > > - enum tree_code *, int *,
> > > > + void *, int *,
> > > > vec<tree> *);
> > > >
> > > > extern unsigned record_stmt_cost (stmt_vector_for_cost *, int, diff
> > > > --git a/gcc/tree.h b/gcc/tree.h index f62c00bc870..346565f84ce
> > > > 100644
> > > > --- a/gcc/tree.h
> > > > +++ b/gcc/tree.h
> > > > @@ -6546,5 +6546,31 @@ extern unsigned fndecl_dealloc_argno (tree);
> > > > if nonnull, set the second argument to the referenced enclosing
> > > > object or pointer. Otherwise return null. */ extern tree
> > > > get_attr_nonstring_decl (tree, tree * = NULL);
> > > > +/* Helper to transparently allow tree codes and builtin function codes
> > > > + exist in one storage entity. */ class code_helper {
> > > > +public:
> > > > + code_helper () {}
> > > > + code_helper (tree_code code) : rep ((int) code) {}
> > > > + code_helper (combined_fn fn) : rep (-(int) fn) {}
> > > > + operator tree_code () const { return is_tree_code () ?
> > > > + (tree_code) rep :
> > > > + ERROR_MARK; }
> > > > + operator combined_fn () const { return is_fn_code () ?
> > > > + (combined_fn) -rep:
> > > > + CFN_LAST; }
> > > > + bool is_tree_code () const { return rep > 0; }
> > > > + bool is_fn_code () const { return rep < 0; }
> > > > + int get_rep () const { return rep; }
> > > > +
> > > > + enum tree_code as_tree_code () const { return is_tree_code () ?
> > > > + (tree_code)* this : MAX_TREE_CODES; } combined_fn as_fn_code
> > > > + () const { return is_fn_code () ? (combined_fn)
> > > *this
> > > > + : CFN_LAST;}
> > > > +
> > > > +private:
> > > > + int rep;
> > > > +};
> > > >
> > > > #endif /* GCC_TREE_H */
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Maxfeldstrasse 5, 90409 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-05-27 13:23 ` Richard Biener
@ 2022-05-31 10:07 ` Joel Hutton
2022-05-31 16:46 ` Tamar Christina
2022-06-01 10:11 ` Richard Biener
0 siblings, 2 replies; 53+ messages in thread
From: Joel Hutton @ 2022-05-31 10:07 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 250 bytes --]
> Can you post an updated patch (after the .cc renaming, and code_helper
> now already moved to tree.h).
>
> Thanks,
> Richard.
Patches attached. They already incorporated the .cc rename, now rebased to be after the change to tree.h
Joel
[-- Attachment #2: 0001-Refactor-to-allow-internal_fn-s.patch --]
[-- Type: application/octet-stream, Size: 26740 bytes --]
From 3467bf531402d83c6427716954b9fab933f858ef Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Wed, 25 Aug 2021 14:31:15 +0100
Subject: [PATCH 1/3] Refactor to allow internal_fn's
Hi all,
This refactor allows widening patterns (such as widen_plus/widen_minus) to be represented as
either internal_fns or tree_codes.
[vect-patterns] Refactor as internal_fn's
Refactor vect-patterns to allow patterns to be internal_fns starting
with widening_plus/minus patterns
gcc/ChangeLog:
* gimple-match.h (class code_helper): Add as_internal_fn, as_tree_code
helper functions.
* gimple.cc (gimple_build): Function to build a GIMPLE_CALL or
GIMPLE_ASSIGN as appropriate, given a code_helper.
* gimple.h (gimple_build): Function prototype.
* tree-core.h (ECF_WIDEN): Flag to mark internal_fn as widening.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Refactor to
use code_helper.
* tree-vect-stmts.cc (vect_gen_widened_results_half): Refactor to
use code_helper.
(vect_create_vectorized_promotion_stmts): Refactor to use
code_helper.
(vectorizable_conversion): Refactor to use code_helper.
gimple_call or gimple_assign.
(supportable_widening_operation): Refactor to use code_helper.
(supportable_narrowing_operation): Refactor to use code_helper.
* tree-vectorizer.h (supportable_widening_operation): Change
prototype to use code_helper.
(supportable_narrowing_operation): change prototype to use
code_helper.
---
gcc/gimple.cc | 24 +++++
gcc/gimple.h | 1 +
gcc/tree-core.h | 3 +
gcc/tree-vect-patterns.cc | 7 +-
gcc/tree-vect-stmts.cc | 216 +++++++++++++++++++++++---------------
gcc/tree-vectorizer.h | 11 +-
gcc/tree.h | 52 +++++++++
7 files changed, 221 insertions(+), 93 deletions(-)
diff --git a/gcc/gimple.cc b/gcc/gimple.cc
index b70ab4d25230374f0c90f93d77f9caf8d57587ee..bd3210cd35298daf7c74276e38658b5151d5cc1f 100644
--- a/gcc/gimple.cc
+++ b/gcc/gimple.cc
@@ -502,6 +502,30 @@ gimple_build_assign (tree lhs, enum tree_code subcode, tree op1 MEM_STAT_DECL)
PASS_MEM_STAT);
}
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple *
+gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
+{
+ if (op0 == NULL_TREE)
+ return NULL;
+ if (ch.is_tree_code ())
+ return op1 == NULL_TREE ? gimple_build_assign (lhs, ch.as_tree_code (),
+ op0) :
+ gimple_build_assign (lhs, ch.as_tree_code (), op0,
+ op1);
+ else
+ {
+ internal_fn fn = as_internal_fn (ch.as_fn_code ());
+ gimple* stmt;
+ if (op1 == NULL_TREE)
+ stmt = gimple_build_call_internal (fn, 1, op0);
+ else
+ stmt = gimple_build_call_internal (fn, 2, op0, op1);
+ gimple_call_set_lhs (stmt, lhs);
+ return stmt;
+ }
+}
/* Build a GIMPLE_COND statement.
diff --git a/gcc/gimple.h b/gcc/gimple.h
index 6b1e89ad74e6b22dd534ff48e48fef688032f844..5350f14f6f29bd8e011b79c2aa79c2bfaef8c58f 100644
--- a/gcc/gimple.h
+++ b/gcc/gimple.h
@@ -1523,6 +1523,7 @@ gcall *gimple_build_call_valist (tree, unsigned, va_list);
gcall *gimple_build_call_internal (enum internal_fn, unsigned, ...);
gcall *gimple_build_call_internal_vec (enum internal_fn, const vec<tree> &);
gcall *gimple_build_call_from_tree (tree, tree);
+gimple* gimple_build (tree, code_helper, tree, tree);
gassign *gimple_build_assign (tree, tree CXX_MEM_STAT_INFO);
gassign *gimple_build_assign (tree, enum tree_code,
tree, tree, tree CXX_MEM_STAT_INFO);
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index ab5fa01e5cb5fb56c1964b93b014ed55a4aa704a..cff6211080bced0bffb39e98039a6550897acf77 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -96,6 +96,9 @@ struct die_struct;
/* Nonzero if this is a cold function. */
#define ECF_COLD (1 << 15)
+/* Nonzero if this is a widening function. */
+#define ECF_WIDEN (1 << 16)
+
/* Call argument flags. */
/* Nonzero if the argument is not used by the function. */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 0fad4dbd0945c6c176f3457b751e812f17fcd148..36e362e1daf3f946c6074600a6a322b3bda67755 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1348,7 +1348,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
- tree_code orig_code, tree_code wide_code,
+ tree_code orig_code, code_helper wide_code,
bool shift_p, const char *name)
{
gimple *last_stmt = last_stmt_info->stmt;
@@ -1391,7 +1391,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
vecctype = get_vectype_for_scalar_type (vinfo, ctype);
}
- enum tree_code dummy_code;
+ code_helper dummy_code;
int dummy_int;
auto_vec<tree> dummy_vec;
if (!vectype
@@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
2, oprnd, half_type, unprom, vectype);
tree var = vect_recog_temp_ssa_var (itype, NULL);
- gimple *pattern_stmt = gimple_build_assign (var, wide_code,
- oprnd[0], oprnd[1]);
+ gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0], oprnd[1]);
if (vecctype != vecitype)
pattern_stmt = vect_convert_output (vinfo, last_stmt_info, ctype,
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 346d8ce280437e00bfeb19a4b4adc59eb96207f9..61b51a29f99bcdf0ff6b4ead4a69163ebf8ed383 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4636,7 +4636,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
STMT_INFO is the original scalar stmt that we are vectorizing. */
static gimple *
-vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
+vect_gen_widened_results_half (vec_info *vinfo, code_helper ch,
tree vec_oprnd0, tree vec_oprnd1, int op_type,
tree vec_dest, gimple_stmt_iterator *gsi,
stmt_vec_info stmt_info)
@@ -4645,14 +4645,12 @@ vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
tree new_temp;
/* Generate half of the widened result: */
- gcc_assert (op_type == TREE_CODE_LENGTH (code));
if (op_type != binary_op)
vec_oprnd1 = NULL;
- new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0, vec_oprnd1);
+ new_stmt = gimple_build (vec_dest, ch, vec_oprnd0, vec_oprnd1);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
-
return new_stmt;
}
@@ -4729,8 +4727,8 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
- enum tree_code code2, int op_type)
+ code_helper ch1,
+ code_helper ch2, int op_type)
{
int i;
tree vop0, vop1, new_tmp1, new_tmp2;
@@ -4746,10 +4744,10 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vop1 = NULL_TREE;
/* Generate the two halves of promotion operation. */
- new_stmt1 = vect_gen_widened_results_half (vinfo, code1, vop0, vop1,
+ new_stmt1 = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
- new_stmt2 = vect_gen_widened_results_half (vinfo, code2, vop0, vop1,
+ new_stmt2 = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
if (is_gimple_call (new_stmt1))
@@ -4846,8 +4844,9 @@ vectorizable_conversion (vec_info *vinfo,
tree scalar_dest;
tree op0, op1 = NULL_TREE;
loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
- enum tree_code code, code1 = ERROR_MARK, code2 = ERROR_MARK;
- enum tree_code codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
+ tree_code tc1;
+ code_helper code, code1, code2;
+ code_helper codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
tree new_temp;
enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type};
int ndts = 2;
@@ -4876,31 +4875,42 @@ vectorizable_conversion (vec_info *vinfo,
&& ! vec_stmt)
return false;
- gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
- if (!stmt)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return false;
- if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
+ if (gimple_get_lhs (stmt) == NULL_TREE ||
+ TREE_CODE(gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- code = gimple_assign_rhs_code (stmt);
- if (!CONVERT_EXPR_CODE_P (code)
- && code != FIX_TRUNC_EXPR
- && code != FLOAT_EXPR
- && code != WIDEN_PLUS_EXPR
- && code != WIDEN_MINUS_EXPR
- && code != WIDEN_MULT_EXPR
- && code != WIDEN_LSHIFT_EXPR)
+ if (TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
- op_type = TREE_CODE_LENGTH (code);
+ bool widen_arith = false;
+ gimple_match_op res_op;
+ if (!gimple_extract_op (stmt, &res_op))
+ return false;
+ code = res_op.code;
+ op_type = res_op.num_ops;
+
+ if (is_gimple_assign (stmt))
+ {
+ widen_arith = (code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR
+ || code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR);
+ }
+ else
+ widen_arith = gimple_call_flags (stmt) & ECF_WIDEN;
+
+ if (!widen_arith
+ && !CONVERT_EXPR_CODE_P (code)
+ && code != FIX_TRUNC_EXPR
+ && code != FLOAT_EXPR)
+ return false;
/* Check types of lhs and rhs. */
- scalar_dest = gimple_assign_lhs (stmt);
+ scalar_dest = gimple_get_lhs (stmt);
lhs_type = TREE_TYPE (scalar_dest);
vectype_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -4938,10 +4948,15 @@ vectorizable_conversion (vec_info *vinfo,
if (op_type == binary_op)
{
- gcc_assert (code == WIDEN_MULT_EXPR || code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR || code == WIDEN_MINUS_EXPR);
+ gcc_assert (code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR
+ || code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR
+ || widen_arith);
+
- op1 = gimple_assign_rhs2 (stmt);
+ op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
+ gimple_call_arg (stmt, 0);
tree vectype1_in;
if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
&op1, &slp_op1, &dt[1], &vectype1_in))
@@ -5025,8 +5040,12 @@ vectorizable_conversion (vec_info *vinfo,
&& code != FLOAT_EXPR
&& !CONVERT_EXPR_CODE_P (code))
return false;
- if (supportable_convert_operation (code, vectype_out, vectype_in, &code1))
+ if (supportable_convert_operation (code.as_tree_code (), vectype_out,
+ vectype_in, &tc1))
+ {
+ code1 = tc1;
break;
+ }
/* FALLTHRU */
unsupported:
if (dump_enabled_p ())
@@ -5037,9 +5056,11 @@ vectorizable_conversion (vec_info *vinfo,
case WIDEN:
if (known_eq (nunits_in, nunits_out))
{
- if (!supportable_half_widening_operation (code, vectype_out,
- vectype_in, &code1))
+ if (!supportable_half_widening_operation (code.as_tree_code (),
+ vectype_out, vectype_in,
+ &tc1))
goto unsupported;
+ code1 = tc1;
gcc_assert (!(multi_step_cvt && op_type == binary_op));
break;
}
@@ -5073,14 +5094,17 @@ vectorizable_conversion (vec_info *vinfo,
if (GET_MODE_SIZE (rhs_mode) == fltsz)
{
- if (!supportable_convert_operation (code, vectype_out,
- cvt_type, &codecvt1))
+ tc1 = ERROR_MARK;
+ if (!supportable_convert_operation (code.as_tree_code (),
+ vectype_out,
+ cvt_type, &tc1))
goto unsupported;
+ codecvt1 = tc1;
}
- else if (!supportable_widening_operation (vinfo, code, stmt_info,
- vectype_out, cvt_type,
- &codecvt1, &codecvt2,
- &multi_step_cvt,
+ else if (!supportable_widening_operation (vinfo, code,
+ stmt_info, vectype_out,
+ cvt_type, &codecvt1,
+ &codecvt2, &multi_step_cvt,
&interm_types))
continue;
else
@@ -5088,8 +5112,9 @@ vectorizable_conversion (vec_info *vinfo,
if (supportable_widening_operation (vinfo, NOP_EXPR, stmt_info,
cvt_type,
- vectype_in, &code1, &code2,
- &multi_step_cvt, &interm_types))
+ vectype_in, &code1,
+ &code2, &multi_step_cvt,
+ &interm_types))
{
found_mode = true;
break;
@@ -5111,10 +5136,14 @@ vectorizable_conversion (vec_info *vinfo,
case NARROW:
gcc_assert (op_type == unary_op);
- if (supportable_narrowing_operation (code, vectype_out, vectype_in,
- &code1, &multi_step_cvt,
+ if (supportable_narrowing_operation (code.as_tree_code (), vectype_out,
+ vectype_in,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
if (code != FIX_TRUNC_EXPR
|| GET_MODE_SIZE (lhs_mode) >= GET_MODE_SIZE (rhs_mode))
@@ -5125,13 +5154,18 @@ vectorizable_conversion (vec_info *vinfo,
cvt_type = get_same_sized_vectype (cvt_type, vectype_in);
if (cvt_type == NULL_TREE)
goto unsupported;
- if (!supportable_convert_operation (code, cvt_type, vectype_in,
- &codecvt1))
+ if (!supportable_convert_operation (code.as_tree_code (), cvt_type,
+ vectype_in,
+ &tc1))
goto unsupported;
+ codecvt1 = tc1;
if (supportable_narrowing_operation (NOP_EXPR, vectype_out, cvt_type,
- &code1, &multi_step_cvt,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
goto unsupported;
default:
@@ -5245,8 +5279,9 @@ vectorizable_conversion (vec_info *vinfo,
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
/* Arguments are ready, create the new vector stmt. */
- gcc_assert (TREE_CODE_LENGTH (code1) == unary_op);
- gassign *new_stmt = gimple_build_assign (vec_dest, code1, vop0);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) code1) == unary_op);
+ gassign *new_stmt = gimple_build_assign (vec_dest,
+ code1.as_tree_code (), vop0);
new_temp = make_ssa_name (vec_dest, new_stmt);
gimple_assign_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
@@ -5278,7 +5313,7 @@ vectorizable_conversion (vec_info *vinfo,
for (i = multi_step_cvt; i >= 0; i--)
{
tree this_dest = vec_dsts[i];
- enum tree_code c1 = code1, c2 = code2;
+ code_helper c1 = code1, c2 = code2;
if (i == 0 && codecvt2 != ERROR_MARK)
{
c1 = codecvt1;
@@ -5288,7 +5323,8 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_half_widening_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
this_dest, gsi,
- c1, op_type);
+ c1.as_tree_code (),
+ op_type);
else
vect_create_vectorized_promotion_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
@@ -5301,9 +5337,11 @@ vectorizable_conversion (vec_info *vinfo,
gimple *new_stmt;
if (cvt_type)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, codecvt1, vop0);
+ new_stmt = gimple_build_assign (new_temp,
+ codecvt1.as_tree_code (),
+ vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -5327,10 +5365,10 @@ vectorizable_conversion (vec_info *vinfo,
if (cvt_type)
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH (((tree_code) codecvt1)) == unary_op);
new_temp = make_ssa_name (vec_dest);
gassign *new_stmt
- = gimple_build_assign (new_temp, codecvt1, vop0);
+ = gimple_build_assign (new_temp, codecvt1.as_tree_code (), vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
vec_oprnds0[i] = new_temp;
}
@@ -5338,7 +5376,7 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_vectorized_demotion_stmts (vinfo, &vec_oprnds0,
multi_step_cvt,
stmt_info, vec_dsts, gsi,
- slp_node, code1);
+ slp_node, code1.as_tree_code ());
break;
}
if (!slp_node)
@@ -11926,9 +11964,11 @@ vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype)
bool
supportable_widening_operation (vec_info *vinfo,
- enum tree_code code, stmt_vec_info stmt_info,
+ code_helper code,
+ stmt_vec_info stmt_info,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, enum tree_code *code2,
+ code_helper *code1,
+ code_helper *code2,
int *multi_step_cvt,
vec<tree> *interm_types)
{
@@ -11939,7 +11979,7 @@ supportable_widening_operation (vec_info *vinfo,
optab optab1, optab2;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
- enum tree_code c1, c2;
+ code_helper c1=MAX_TREE_CODES, c2=MAX_TREE_CODES;
int i;
tree prev_type, intermediate_type;
machine_mode intermediate_mode, prev_mode;
@@ -11949,7 +11989,7 @@ supportable_widening_operation (vec_info *vinfo,
if (loop_info)
vect_loop = LOOP_VINFO_LOOP (loop_info);
- switch (code)
+ switch (code.as_tree_code ())
{
case WIDEN_MULT_EXPR:
/* The result of a vectorized widening operation usually requires
@@ -11990,8 +12030,9 @@ supportable_widening_operation (vec_info *vinfo,
&& !nested_in_vect_loop_p (vect_loop, stmt_info)
&& supportable_widening_operation (vinfo, VEC_WIDEN_MULT_EVEN_EXPR,
stmt_info, vectype_out,
- vectype_in, code1, code2,
- multi_step_cvt, interm_types))
+ vectype_in, code1,
+ code2, multi_step_cvt,
+ interm_types))
{
/* Elements in a vector with vect_used_by_reduction property cannot
be reordered if the use chain with this property does not have the
@@ -12054,6 +12095,9 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_UNPACK_FIX_TRUNC_HI_EXPR;
break;
+ case MAX_TREE_CODES:
+ break;
+
default:
gcc_unreachable ();
}
@@ -12061,13 +12105,16 @@ supportable_widening_operation (vec_info *vinfo,
if (BYTES_BIG_ENDIAN && c1 != VEC_WIDEN_MULT_EVEN_EXPR)
std::swap (c1, c2);
+
if (code == FIX_TRUNC_EXPR)
{
/* The signedness is determined from output operand. */
- optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
- optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ optab1 = optab_for_tree_code (c1.as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.as_tree_code (), vectype_out,
+ optab_default);
}
- else if (CONVERT_EXPR_CODE_P (code)
+ else if (CONVERT_EXPR_CODE_P (code.as_tree_code ())
&& VECTOR_BOOLEAN_TYPE_P (wide_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
&& TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
@@ -12080,8 +12127,8 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ optab1 = optab_for_tree_code (c1.as_tree_code (), vectype, optab_default);
+ optab2 = optab_for_tree_code (c2.as_tree_code (), vectype, optab_default);
}
if (!optab1 || !optab2)
@@ -12092,8 +12139,12 @@ supportable_widening_operation (vec_info *vinfo,
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- *code1 = c1;
- *code2 = c2;
+ if (code.is_tree_code ())
+ {
+ *code1 = c1;
+ *code2 = c2;
+ }
+
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
@@ -12114,7 +12165,7 @@ supportable_widening_operation (vec_info *vinfo,
prev_type = vectype;
prev_mode = vec_mode;
- if (!CONVERT_EXPR_CODE_P (code))
+ if (!CONVERT_EXPR_CODE_P ((tree_code) code))
return false;
/* We assume here that there will not be more than MAX_INTERM_CVT_STEPS
@@ -12145,8 +12196,10 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab3 = optab_for_tree_code (c1, intermediate_type, optab_default);
- optab4 = optab_for_tree_code (c2, intermediate_type, optab_default);
+ optab3 = optab_for_tree_code (c1.as_tree_code (), intermediate_type,
+ optab_default);
+ optab4 = optab_for_tree_code (c2.as_tree_code (), intermediate_type,
+ optab_default);
}
if (!optab3 || !optab4
@@ -12181,7 +12234,6 @@ supportable_widening_operation (vec_info *vinfo,
return false;
}
-
/* Function supportable_narrowing_operation
Check whether an operation represented by the code CODE is a
@@ -12205,7 +12257,7 @@ supportable_widening_operation (vec_info *vinfo,
bool
supportable_narrowing_operation (enum tree_code code,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, int *multi_step_cvt,
+ tree_code* _code1, int *multi_step_cvt,
vec<tree> *interm_types)
{
machine_mode vec_mode;
@@ -12217,8 +12269,8 @@ supportable_narrowing_operation (enum tree_code code,
tree intermediate_type, prev_type;
machine_mode intermediate_mode, prev_mode;
int i;
- unsigned HOST_WIDE_INT n_elts;
bool uns;
+ tree_code * code1 = (tree_code*) _code1;
*multi_step_cvt = 0;
switch (code)
@@ -12227,9 +12279,8 @@ supportable_narrowing_operation (enum tree_code code,
c1 = VEC_PACK_TRUNC_EXPR;
if (VECTOR_BOOLEAN_TYPE_P (narrow_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
- && SCALAR_INT_MODE_P (TYPE_MODE (vectype))
- && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&n_elts)
- && n_elts < BITS_PER_UNIT)
+ && TYPE_MODE (narrow_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
optab1 = vec_pack_sbool_trunc_optab;
else
optab1 = optab_for_tree_code (c1, vectype, optab_default);
@@ -12320,9 +12371,8 @@ supportable_narrowing_operation (enum tree_code code,
= lang_hooks.types.type_for_mode (intermediate_mode, uns);
if (VECTOR_BOOLEAN_TYPE_P (intermediate_type)
&& VECTOR_BOOLEAN_TYPE_P (prev_type)
- && SCALAR_INT_MODE_P (prev_mode)
- && TYPE_VECTOR_SUBPARTS (intermediate_type).is_constant (&n_elts)
- && n_elts < BITS_PER_UNIT)
+ && intermediate_mode == prev_mode
+ && SCALAR_INT_MODE_P (prev_mode))
interm_optab = vec_pack_sbool_trunc_optab;
else
interm_optab
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 642eb0aeb21264cd736a479b1ec25357abef29cd..50ff8eeac1e6b9859302bd784f10ee3d8ff4b4dc 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2120,13 +2120,12 @@ extern bool vect_is_simple_use (vec_info *, stmt_vec_info, slp_tree,
enum vect_def_type *,
tree *, stmt_vec_info * = NULL);
extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
-extern bool supportable_widening_operation (vec_info *,
- enum tree_code, stmt_vec_info,
- tree, tree, enum tree_code *,
- enum tree_code *, int *,
- vec<tree> *);
+extern bool supportable_widening_operation (vec_info*, code_helper,
+ stmt_vec_info, tree, tree,
+ code_helper*, code_helper*,
+ int*, vec<tree> *);
extern bool supportable_narrowing_operation (enum tree_code, tree, tree,
- enum tree_code *, int *,
+ tree_code *, int *,
vec<tree> *);
extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
diff --git a/gcc/tree.h b/gcc/tree.h
index f84958933d51144bb6ce7cc41eca5f7f06814550..e51e34c051d9b91d1c02a4b2fefdb2b15606a36f 100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -92,6 +92,10 @@ public:
bool is_fn_code () const { return rep < 0; }
bool is_internal_fn () const;
bool is_builtin_fn () const;
+ enum tree_code as_tree_code () const { return is_tree_code () ?
+ (tree_code)* this : MAX_TREE_CODES; }
+ combined_fn as_fn_code () const { return is_fn_code () ? (combined_fn) *this
+ : CFN_LAST;}
int get_rep () const { return rep; }
bool operator== (const code_helper &other) { return rep == other.rep; }
bool operator!= (const code_helper &other) { return rep != other.rep; }
@@ -6657,6 +6661,54 @@ extern unsigned fndecl_dealloc_argno (tree);
if nonnull, set the second argument to the referenced enclosing
object or pointer. Otherwise return null. */
extern tree get_attr_nonstring_decl (tree, tree * = NULL);
+/* Helper to transparently allow tree codes and builtin function codes
+ exist in one storage entity. */
+class code_helper
+{
+public:
+ code_helper () {}
+ code_helper (tree_code code) : rep ((int) code) {}
+ code_helper (combined_fn fn) : rep (-(int) fn) {}
+ code_helper (internal_fn fn) : rep (-(int) as_combined_fn (fn)) {}
+ explicit operator tree_code () const { return (tree_code) rep; }
+ explicit operator combined_fn () const { return (combined_fn) -rep; }
+ explicit operator internal_fn () const;
+ explicit operator built_in_function () const;
+ bool is_tree_code () const { return rep > 0; }
+ bool is_fn_code () const { return rep < 0; }
+ bool is_internal_fn () const;
+ bool is_builtin_fn () const;
+ int get_rep () const { return rep; }
+ bool operator== (const code_helper &other) { return rep == other.rep; }
+ bool operator!= (const code_helper &other) { return rep != other.rep; }
+ bool operator== (tree_code c) { return rep == code_helper (c).rep; }
+ bool operator!= (tree_code c) { return rep != code_helper (c).rep; }
+
+private:
+ int rep;
+};
+
+inline code_helper::operator internal_fn () const
+{
+ return as_internal_fn (combined_fn (*this));
+}
+
+inline code_helper::operator built_in_function () const
+{
+ return as_builtin_fn (combined_fn (*this));
+}
+
+inline bool
+code_helper::is_internal_fn () const
+{
+ return is_fn_code () && internal_fn_p (combined_fn (*this));
+}
+
+inline bool
+code_helper::is_builtin_fn () const
+{
+ return is_fn_code () && builtin_fn_p (combined_fn (*this));
+}
extern int get_target_clone_attr_len (tree);
--
2.17.1
[-- Attachment #3: 0002-Refactor-widen_plus-as-internal_fn.patch --]
[-- Type: application/octet-stream, Size: 22720 bytes --]
From 70fa5fc3e8282a73b973bdd79fccd3450d5b312c Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Wed, 26 Jan 2022 14:00:17 +0000
Subject: [PATCH 2/3] Refactor widen_plus as internal_fn
This patch replaces the existing tree_code widen_plus and widen_minus
patterns with internal_fn versions.
DEF_INTERNAL_OPTAB_MULTI_FN is like DEF_INTERNAL_OPTAB_FN except it provides convenience wrappers for defining conversions that require a hi/lo split, like widening and narrowing operations. Each definition for <NAME> will require an optab named <OPTAB> and two other optabs that you specify for signed and unsigned. The hi/lo pair is necessary because the widening operations take n narrow elements as inputs and return n/2 wide elements as outputs. The 'lo' operation operates on the first n/2 elements of input. The 'hi' operation operates on the second n/2 elements of input. Defining an internal_fn along with hi/lo variations allows a single internal function to be returned from a vect_recog function that will later be expanded to hi/lo.
DEF_INTERNAL_OPTAB_MULTI_FN is used in internal-fn.def to register a widening internal_fn. It is defined differently in different places and internal-fn.def is sourced from those places so the parameters given can be reused.
internal-fn.c: defined to expand to hi/lo signed/unsigned optabs, later defined to generate the 'expand_' functions for the hi/lo versions of the fn.
internal-fn.def: defined to invoke DEF_INTERNAL_OPTAB_FN for the original and hi/lo variants of the internal_fn
For example:
IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>addl_hi_<mode> -> (u/s)addl2
IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>addl_lo_<mode> -> (u/s)addl
This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
gcc/ChangeLog:
2022-04-13 Joel Hutton <joel.hutton@arm.com>
2022-04-13 Tamar Christina <tamar.christina@arm.com>
* internal-fn.cc (INCLUDE_MAP): Include maps for use in optab
lookup.
(DEF_INTERNAL_OPTAB_MULTI_FN): Macro to define an internal_fn that
expands into multiple internal_fns (for widening).
(ifn_cmp): Function to compare ifn's for sorting/searching.
(lookup_multi_ifn_optab): Add lookup function.
(lookup_multi_internal_fn): Add lookup function.
(commutative_binary_fn_p): Add widen_plus fn's.
* internal-fn.def (DEF_INTERNAL_OPTAB_MULTI_FN): Define widening
plus,minus functions.
(VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS tree code.
(VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS tree code.
* internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
(lookup_multi_ifn_optab): Add prototype.
(lookup_multi_internal_fn): Add prototype.
* optabs.cc (commutative_optab_p): Add widening plus, minus optabs.
* optabs.def (OPTAB_CD): widen add, sub optabs
* tree-core.h (ECF_MULTI): Flag to indicate if a function decays
into hi/lo parts.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
patterns with a hi/lo split.
(vect_recog_widen_plus_pattern): Refactor to return
IFN_VECT_WIDEN_PLUS.
(vect_recog_widen_minus_pattern): Refactor to return new
IFN_VEC_WIDEN_MINUS.
* tree-vect-stmts.cc (vectorizable_conversion): Add widen plus/minus
ifn
support.
(supportable_widening_operation): Add widen plus/minus ifn support.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-widen-add.c: Test that new
IFN_VEC_WIDEN_PLUS is being used.
* gcc.target/aarch64/vect-widen-sub.c: Test that new
IFN_VEC_WIDEN_MINUS is being used.
---
gcc/internal-fn.cc | 107 ++++++++++++++++++
gcc/internal-fn.def | 23 ++++
gcc/internal-fn.h | 7 ++
gcc/optabs.cc | 12 +-
gcc/optabs.def | 2 +
.../gcc.target/aarch64/vect-widen-add.c | 4 +-
.../gcc.target/aarch64/vect-widen-sub.c | 4 +-
gcc/tree-core.h | 4 +
gcc/tree-vect-patterns.cc | 37 ++++--
gcc/tree-vect-stmts.cc | 60 +++++++++-
10 files changed, 244 insertions(+), 16 deletions(-)
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 8b1733e20c4455e4e8c383c92fe859f4256cae69..e95b13af884f67990ad43c286990a351e2bd641b 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -17,6 +17,7 @@ You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3. If not see
<http://www.gnu.org/licenses/>. */
+#define INCLUDE_MAP
#include "config.h"
#include "system.h"
#include "coretypes.h"
@@ -70,6 +71,26 @@ const int internal_fn_flags_array[] = {
0
};
+const enum internal_fn internal_fn_hilo_keys_array[] = {
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ IFN_##NAME##_LO, \
+ IFN_##NAME##_HI,
+#include "internal-fn.def"
+ IFN_LAST
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+};
+
+const optab internal_fn_hilo_values_array[] = {
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ SOPTAB##_lo_optab, UOPTAB##_lo_optab, \
+ SOPTAB##_hi_optab, UOPTAB##_hi_optab,
+#include "internal-fn.def"
+ unknown_optab, unknown_optab
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+};
+
/* Return the internal function called NAME, or IFN_LAST if there's
no such function. */
@@ -90,6 +111,62 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+static int
+ifn_cmp (const void *a_, const void *b_)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ auto *a = (const std::pair<ifn_pair, optab> *)a_;
+ auto *b = (const std::pair<ifn_pair, optab> *)b_;
+ return (int) (a->first.first) - (b->first.first);
+}
+
+/* Return the optab belonging to the given internal function NAME for the given
+ SIGN or unknown_optab. */
+
+optab
+lookup_multi_ifn_optab (enum internal_fn fn, unsigned sign)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ typedef auto_vec <std::pair<ifn_pair, optab>>fn_to_optab_map_type;
+ static fn_to_optab_map_type *fn_to_optab_map;
+
+ if (!fn_to_optab_map)
+ {
+ unsigned num
+ = sizeof (internal_fn_hilo_keys_array) / sizeof (enum internal_fn);
+ fn_to_optab_map = new fn_to_optab_map_type ();
+ for (unsigned int i = 0; i < num - 1; ++i)
+ {
+ enum internal_fn fn = internal_fn_hilo_keys_array[i];
+ optab v1 = internal_fn_hilo_values_array[2*i];
+ optab v2 = internal_fn_hilo_values_array[2*i + 1];
+ ifn_pair key1 (fn, 0);
+ fn_to_optab_map->safe_push ({key1, v1});
+ ifn_pair key2 (fn, 1);
+ fn_to_optab_map->safe_push ({key2, v2});
+ }
+ fn_to_optab_map->qsort(ifn_cmp);
+ }
+
+ ifn_pair new_pair (fn, sign ? 1 : 0);
+ optab tmp;
+ std::pair<ifn_pair,optab> pair_wrap (new_pair, tmp);
+ auto entry = fn_to_optab_map->bsearch (&pair_wrap, ifn_cmp);
+ return entry != fn_to_optab_map->end () ? entry->second : unknown_optab;
+}
+
+extern void
+lookup_multi_internal_fn (enum internal_fn ifn, enum internal_fn *lo,
+ enum internal_fn *hi)
+{
+ int ecf_flags = internal_fn_flags (ifn);
+ gcc_assert (ecf_flags & ECF_MULTI);
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -3906,6 +3983,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -3991,6 +4071,32 @@ set_edom_supported_p (void)
#endif
}
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(CODE, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ static void \
+ expand_##CODE (internal_fn, gcall *) \
+ { \
+ gcc_unreachable (); \
+ } \
+ static void \
+ expand_##CODE##_LO (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_lo##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_lo##_optab); \
+ } \
+ static void \
+ expand_##CODE##_HI (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_hi##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_hi##_optab); \
+ }
+
#define DEF_INTERNAL_OPTAB_FN(CODE, FLAGS, OPTAB, TYPE) \
static void \
expand_##CODE (internal_fn fn, gcall *stmt) \
@@ -4007,6 +4113,7 @@ set_edom_supported_p (void)
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index d2d550d358606022b1cb44fa842f06e0be507bc3..4635a9c8af9ad27bb05d7510388d0fe2270428e5 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -82,6 +82,13 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_OPTAB_MULTI_FN is like DEF_INTERNAL_OPTAB_FN except it
+ provides convenience wrappers for defining conversions that require a
+ hi/lo split, like widening and narrowing operations. Each definition
+ for <NAME> will require an optab named <OPTAB> and two other optabs that
+ you specify for signed and unsigned.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -120,6 +127,14 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME, FLAGS | ECF_MULTI, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, unknown, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, unknown, TYPE)
+#endif
+
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -292,6 +307,14 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_OPTAB_MULTI_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_WIDEN | ECF_NOTHROW,
+ vec_widen_add, vec_widen_saddl, vec_widen_uaddl,
+ binary)
+DEF_INTERNAL_OPTAB_MULTI_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_WIDEN | ECF_NOTHROW,
+ vec_widen_sub, vec_widen_ssubl, vec_widen_usubl,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 23c014a963c4d72da92c763db87ee486a2adb485..b35de19747d251d19dc13de1e0323368bd2ebdf2 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,9 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern optab lookup_multi_ifn_optab (enum internal_fn, unsigned);
+extern void lookup_multi_internal_fn (enum internal_fn, enum internal_fn *,
+ enum internal_fn *);
/* Return the ECF_* flags for function FN. */
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index c0a68471d2ddf08bc0e6a3fd592ebb9f05e516c1..7e904b3e154d018779bb1a36de74e6997f70e193 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,17 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_add_optab
+ || binoptab == vec_widen_sub_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_ssubl_hi_optab
+ || binoptab == vec_widen_ssubl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab
+ || binoptab == vec_widen_usubl_hi_optab
+ || binoptab == vec_widen_usubl_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 801310ebaa7d469520809bb7efed6820f8eb866b..a7881dcb49e4ef07d8f07aa31214eb3a7a944e2e 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -78,6 +78,8 @@ OPTAB_CD(smsub_widen_optab, "msub$b$a4")
OPTAB_CD(umsub_widen_optab, "umsub$b$a4")
OPTAB_CD(ssmsub_widen_optab, "ssmsub$b$a4")
OPTAB_CD(usmsub_widen_optab, "usmsub$a$b4")
+OPTAB_CD(vec_widen_add_optab, "add$a$b3")
+OPTAB_CD(vec_widen_sub_optab, "sub$a$b3")
OPTAB_CD(vec_load_lanes_optab, "vec_load_lanes$a$b")
OPTAB_CD(vec_store_lanes_optab, "vec_store_lanes$a$b")
OPTAB_CD(vec_mask_load_lanes_optab, "vec_mask_load_lanes$a$b")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index cff6211080bced0bffb39e98039a6550897acf77..d0c8b812cfb9c3ac83bf25fff0431b08cb7d823d 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -99,6 +99,10 @@ struct die_struct;
/* Nonzero if this is a widening function. */
#define ECF_WIDEN (1 << 16)
+/* Nonzero if this is a function that decomposes into a lo/hi operation. */
+#define ECF_MULTI (1 << 17)
+
+
/* Call argument flags. */
/* Nonzero if the argument is not used by the function. */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 36e362e1daf3f946c6074600a6a322b3bda67755..0fd587da12b4a17b238327ae60f5a2a7a0efc514 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1349,14 +1349,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1422,6 +1424,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1435,26 +1451,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_popcount_pattern
@@ -5618,6 +5638,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 61b51a29f99bcdf0ff6b4ead4a69163ebf8ed383..c31831df723eeae8ea4fca2790a18b562106c889 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4880,7 +4880,7 @@ vectorizable_conversion (vec_info *vinfo,
return false;
if (gimple_get_lhs (stmt) == NULL_TREE ||
- TREE_CODE(gimple_get_lhs (stmt)) != SSA_NAME)
+ TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
if (TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
@@ -12125,12 +12125,62 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
- {
- optab1 = optab_for_tree_code (c1.as_tree_code (), vectype, optab_default);
- optab2 = optab_for_tree_code (c2.as_tree_code (), vectype, optab_default);
+
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn (code.as_fn_code ());
+ int ecf_flags = internal_fn_flags (ifn);
+ gcc_assert (ecf_flags & ECF_MULTI);
+
+ switch (code.as_fn_code ())
+ {
+ case CFN_VEC_WIDEN_PLUS:
+ break;
+ case CFN_VEC_WIDEN_MINUS:
+ break;
+ case CFN_LAST:
+ default:
+ return false;
+ }
+
+ internal_fn lo, hi;
+ lookup_multi_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
}
+ if (code.is_tree_code ())
+ {
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1.as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.as_tree_code (), vectype_out,
+ optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P (code.as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1.as_tree_code (), vectype,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.as_tree_code (), vectype,
+ optab_default);
+ }
+ }
+
if (!optab1 || !optab2)
return false;
--
2.17.1
[-- Attachment #4: 0003-Remove-widen_plus-minus_expr-tree-codes.patch --]
[-- Type: application/octet-stream, Size: 19457 bytes --]
From 40224aa09ccd4f44aa6bd843f6c7ce0dbb3b6970 Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Fri, 28 Jan 2022 12:04:44 +0000
Subject: [PATCH 3/3] Remove widen_plus/minus_expr tree codes
This patch removes the old widen plus/minus tree codes which have been
replaced by internal functions.
gcc/ChangeLog:
* doc/generic.texi: Remove old tree codes.
* expr.cc (expand_expr_real_2): Remove old tree code cases.
* gimple-pretty-print.cc (dump_binary_rhs): Remove old tree code
cases.
* optabs-tree.cc (optab_for_tree_code): Remove old tree code cases.
(supportable_half_widening_operation): Remove old tree code cases.
* tree-cfg.cc (verify_gimple_assign_binary): Remove old tree code
cases.
* tree-inline.cc (estimate_operator_cost): Remove old tree code
cases.
* tree-pretty-print.cc (dump_generic_node): Remove tree code definition.
(op_symbol_code): Remove old tree code
cases.
* tree-vect-data-refs.cc (vect_get_smallest_scalar_type): Remove old tree code
cases.
(vect_analyze_data_ref_accesses): Remove old tree code
cases.
* tree-vect-generic.cc (expand_vector_operations_1): Remove old tree code
cases.
* tree-vect-patterns.cc (vect_widened_op_tree): Refactor ot replace
usage in vect_recog_sad_pattern.
(vect_recog_sad_pattern): Replace tree code widening pattern with
internal function.
(vect_recog_average_pattern): Replace tree code widening pattern
with internal function.
* tree-vect-stmts.cc (vectorizable_conversion): Remove old tree code
cases.
(supportable_widening_operation): Remove old tree code
cases.
* tree.def (WIDEN_PLUS_EXPR): Remove tree code definition.
(WIDEN_MINUS_EXPR): Remove tree code definition.
(VEC_WIDEN_PLUS_HI_EXPR): Remove tree code definition.
(VEC_WIDEN_PLUS_LO_EXPR): Remove tree code definition.
(VEC_WIDEN_MINUS_HI_EXPR): Remove tree code definition.
(VEC_WIDEN_MINUS_LO_EXPR): Remove tree code definition.
---
gcc/doc/generic.texi | 31 -------------------------------
gcc/expr.cc | 6 ------
gcc/gimple-pretty-print.cc | 4 ----
gcc/optabs-tree.cc | 24 ------------------------
gcc/tree-cfg.cc | 6 ------
gcc/tree-inline.cc | 6 ------
gcc/tree-pretty-print.cc | 12 ------------
gcc/tree-vect-data-refs.cc | 8 +++-----
gcc/tree-vect-generic.cc | 4 ----
gcc/tree-vect-patterns.cc | 36 +++++++++++++++++++++++++-----------
gcc/tree-vect-stmts.cc | 18 ++----------------
gcc/tree.def | 6 ------
12 files changed, 30 insertions(+), 131 deletions(-)
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index e5f9d1be8ea81f3da002ec3bb925590d331a2551..344045efd419b0cc3a11771acf70d2fd279c48ac 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,10 +1811,6 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1861,33 +1857,6 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
-@item VEC_WIDEN_PLUS_HI_EXPR
-@itemx VEC_WIDEN_PLUS_LO_EXPR
-These nodes represent widening vector addition of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The result
-is a vector that contains half as many elements, of an integral type whose size
-is twice as wide. In the case of @code{VEC_WIDEN_PLUS_HI_EXPR} the high
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products. In the case of @code{VEC_WIDEN_PLUS_LO_EXPR} the low
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products.
-
-@item VEC_WIDEN_MINUS_HI_EXPR
-@itemx VEC_WIDEN_MINUS_LO_EXPR
-These nodes represent widening vector subtraction of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The high/low
-elements of the second vector are subtracted from the high/low elements of the
-first. The result is a vector that contains half as many elements, of an
-integral type whose size is twice as wide. In the case of
-@code{VEC_WIDEN_MINUS_HI_EXPR} the high @code{N/2} elements of the second
-vector are subtracted from the high @code{N/2} of the first to produce the
-vector of @code{N/2} products. In the case of
-@code{VEC_WIDEN_MINUS_LO_EXPR} the low @code{N/2} elements of the second
-vector are subtracted from the low @code{N/2} of the first to produce the
-vector of @code{N/2} products.
-
@item VEC_UNPACK_HI_EXPR
@itemx VEC_UNPACK_LO_EXPR
These nodes represent unpacking of the high and low parts of the input vector,
diff --git a/gcc/expr.cc b/gcc/expr.cc
index 7197996cec7d24dd43d60928d5618b32b77677a1..1f941efc9e995c7f6a35ff93aaa6bd3c35faaa1f 100644
--- a/gcc/expr.cc
+++ b/gcc/expr.cc
@@ -9337,8 +9337,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
target, unsignedp);
return target;
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_MULT_EXPR:
/* If first operand is constant, swap them.
Thus the following special case checks need only
@@ -10116,10 +10114,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
return temp;
}
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/gimple-pretty-print.cc b/gcc/gimple-pretty-print.cc
index ebd87b20a0adc080c4a8f9429e75f49b96e72f9a..2a1a5b7f811ca341e8ee7e85a9701d3a37ff80bf 100644
--- a/gcc/gimple-pretty-print.cc
+++ b/gcc/gimple-pretty-print.cc
@@ -459,10 +459,6 @@ dump_binary_rhs (pretty_printer *buffer, const gassign *gs, int spc,
case VEC_PACK_FLOAT_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_SERIES_EXPR:
for (p = get_tree_code_name (code); *p; p++)
pp_character (buffer, TOUPPER (*p));
diff --git a/gcc/optabs-tree.cc b/gcc/optabs-tree.cc
index 8383fe820b080f6d66f83dcf3b77d3c9f869f4bc..2f5f93dc6624f86f6b5618cf6e7aa2b508053a64 100644
--- a/gcc/optabs-tree.cc
+++ b/gcc/optabs-tree.cc
@@ -190,22 +190,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
return (TYPE_UNSIGNED (type)
? vec_widen_ushiftl_lo_optab : vec_widen_sshiftl_lo_optab);
- case VEC_WIDEN_PLUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_lo_optab : vec_widen_saddl_lo_optab);
-
- case VEC_WIDEN_PLUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_hi_optab : vec_widen_saddl_hi_optab);
-
- case VEC_WIDEN_MINUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_lo_optab : vec_widen_ssubl_lo_optab);
-
- case VEC_WIDEN_MINUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_hi_optab : vec_widen_ssubl_hi_optab);
-
case VEC_UNPACK_HI_EXPR:
return (TYPE_UNSIGNED (type)
? vec_unpacku_hi_optab : vec_unpacks_hi_optab);
@@ -312,8 +296,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
'hi'/'lo' pair using codes such as VEC_WIDEN_MINUS_HI/LO.
Supported widening operations:
- WIDEN_MINUS_EXPR
- WIDEN_PLUS_EXPR
WIDEN_MULT_EXPR
WIDEN_LSHIFT_EXPR
@@ -345,12 +327,6 @@ supportable_half_widening_operation (enum tree_code code, tree vectype_out,
case WIDEN_LSHIFT_EXPR:
*code1 = LSHIFT_EXPR;
break;
- case WIDEN_MINUS_EXPR:
- *code1 = MINUS_EXPR;
- break;
- case WIDEN_PLUS_EXPR:
- *code1 = PLUS_EXPR;
- break;
case WIDEN_MULT_EXPR:
*code1 = MULT_EXPR;
break;
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index 8de1b144a426776bf464765477c71ee8f2e52b81..46eed1e1f22052fc077f2fc25e5be627bce541b6 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -3948,8 +3948,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case PLUS_EXPR:
case MINUS_EXPR:
{
@@ -4070,10 +4068,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 043e1d5987a4c4b0159109dafb85a805ca828c1e..c0bebb7f4de36838341ed62389ad0e2b79f03034 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4288,8 +4288,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case REALIGN_LOAD_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case DOT_PROD_EXPR:
@@ -4298,10 +4296,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case WIDEN_MULT_MINUS_EXPR:
case WIDEN_LSHIFT_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-pretty-print.cc b/gcc/tree-pretty-print.cc
index 6acd394a0790ad2ad989f195a3288f0f0a8cc489..53ca62dc1a6873ae9365f199061bde9edd486196 100644
--- a/gcc/tree-pretty-print.cc
+++ b/gcc/tree-pretty-print.cc
@@ -2825,8 +2825,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
break;
/* Binary arithmetic and logic expressions. */
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case MULT_EXPR:
@@ -3790,10 +3788,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
case VEC_SERIES_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
@@ -4311,12 +4305,6 @@ op_symbol_code (enum tree_code code)
case WIDEN_LSHIFT_EXPR:
return "w<<";
- case WIDEN_PLUS_EXPR:
- return "w+";
-
- case WIDEN_MINUS_EXPR:
- return "w-";
-
case POINTER_PLUS_EXPR:
return "+";
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index d20a10a1524164eef788ab4b88ba57c7a09c3387..98dd56ff022233ccead36a1f5a5e896e352f9f5b 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -136,8 +136,6 @@ vect_get_smallest_scalar_type (stmt_vec_info stmt_info, tree scalar_type)
|| gimple_assign_rhs_code (assign) == WIDEN_SUM_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_MULT_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_LSHIFT_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_PLUS_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_MINUS_EXPR
|| gimple_assign_rhs_code (assign) == FLOAT_EXPR)
{
tree rhs_type = TREE_TYPE (gimple_assign_rhs1 (assign));
@@ -3172,8 +3170,8 @@ vect_analyze_data_ref_accesses (vec_info *vinfo,
break;
/* Check that the DR_INITs are compile-time constants. */
- if (!tree_fits_shwi_p (DR_INIT (dra))
- || !tree_fits_shwi_p (DR_INIT (drb)))
+ if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST
+ || TREE_CODE (DR_INIT (drb)) != INTEGER_CST)
break;
/* Different .GOMP_SIMD_LANE calls still give the same lane,
@@ -3225,7 +3223,7 @@ vect_analyze_data_ref_accesses (vec_info *vinfo,
unsigned HOST_WIDE_INT step
= absu_hwi (tree_to_shwi (DR_STEP (dra)));
if (step != 0
- && step <= ((unsigned HOST_WIDE_INT)init_b - init_a))
+ && step <= (unsigned HOST_WIDE_INT)(init_b - init_a))
break;
}
}
diff --git a/gcc/tree-vect-generic.cc b/gcc/tree-vect-generic.cc
index 92aba5d4af61dd478ec3f1b94854e4ad84166774..5823b08baf70b89b22ecc148b0702a84671ad084 100644
--- a/gcc/tree-vect-generic.cc
+++ b/gcc/tree-vect-generic.cc
@@ -2209,10 +2209,6 @@ expand_vector_operations_1 (gimple_stmt_iterator *gsi,
arguments, not the widened result. VEC_UNPACK_FLOAT_*_EXPR is
calculated in the same way above. */
if (code == WIDEN_SUM_EXPR
- || code == VEC_WIDEN_PLUS_HI_EXPR
- || code == VEC_WIDEN_PLUS_LO_EXPR
- || code == VEC_WIDEN_MINUS_HI_EXPR
- || code == VEC_WIDEN_MINUS_LO_EXPR
|| code == VEC_WIDEN_MULT_HI_EXPR
|| code == VEC_WIDEN_MULT_LO_EXPR
|| code == VEC_WIDEN_MULT_EVEN_EXPR
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 0fd587da12b4a17b238327ae60f5a2a7a0efc514..0d821413b971d983c6562c3e8fbe60e2c3d0cb94 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -557,21 +557,29 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else
+ rhs_code = gimple_call_combined_fn (stmt);
+
+ if (rhs_code.as_tree_code () != code
+ && rhs_code.get_rep () != widened_code.get_rep ())
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = is_gimple_assign (stmt) ? gimple_assign_lhs (stmt):
+ gimple_call_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -584,7 +592,11 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op;
+ if (is_gimple_assign (stmt))
+ op = gimple_op (stmt, i + 1);
+ else
+ op = gimple_call_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1297,8 +1309,9 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
- false, 2, unprom, &half_type))
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ CFN_VEC_WIDEN_MINUS, false, 2, unprom,
+ &half_type))
return NULL;
vect_pattern_detected ("vect_recog_sad_pattern", last_stmt);
@@ -2335,9 +2348,10 @@ vect_recog_average_pattern (vec_info *vinfo,
internal_fn ifn = IFN_AVG_FLOOR;
vect_unpromoted_value unprom[3];
tree new_type;
+ enum optab_subtype subtype;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
- unprom, &new_type);
+ CFN_VEC_WIDEN_PLUS, false, 3,
+ unprom, &new_type, &subtype);
if (nops == 0)
return NULL;
if (nops == 3)
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index c31831df723eeae8ea4fca2790a18b562106c889..9adbb9fbf116ef316d5bed2c84a7074722055717 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4895,9 +4895,7 @@ vectorizable_conversion (vec_info *vinfo,
if (is_gimple_assign (stmt))
{
- widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
+ widen_arith = (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR);
}
else
@@ -4950,8 +4948,6 @@ vectorizable_conversion (vec_info *vinfo,
{
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
|| widen_arith);
@@ -11976,7 +11972,7 @@ supportable_widening_operation (vec_info *vinfo,
class loop *vect_loop = NULL;
machine_mode vec_mode;
enum insn_code icode1, icode2;
- optab optab1, optab2;
+ optab optab1 = unknown_optab, optab2 = unknown_optab;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
code_helper c1=MAX_TREE_CODES, c2=MAX_TREE_CODES;
@@ -12070,16 +12066,6 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_WIDEN_LSHIFT_HI_EXPR;
break;
- case WIDEN_PLUS_EXPR:
- c1 = VEC_WIDEN_PLUS_LO_EXPR;
- c2 = VEC_WIDEN_PLUS_HI_EXPR;
- break;
-
- case WIDEN_MINUS_EXPR:
- c1 = VEC_WIDEN_MINUS_LO_EXPR;
- c2 = VEC_WIDEN_MINUS_HI_EXPR;
- break;
-
CASE_CONVERT:
c1 = VEC_UNPACK_LO_EXPR;
c2 = VEC_UNPACK_HI_EXPR;
diff --git a/gcc/tree.def b/gcc/tree.def
index 62650b6934b337c5d56e5393dc114173d72c9aa9..9b2dce3576440c445d3240b9ed937fe67c9a5992 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1383,8 +1383,6 @@ DEFTREECODE (WIDEN_MULT_MINUS_EXPR, "widen_mult_minus_expr", tcc_expression, 3)
the first argument from type t1 to type t2, and then shifting it
by the second argument. */
DEFTREECODE (WIDEN_LSHIFT_EXPR, "widen_lshift_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_PLUS_EXPR, "widen_plus_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_MINUS_EXPR, "widen_minus_expr", tcc_binary, 2)
/* Widening vector multiplication.
The two operands are vectors with N elements of size S. Multiplying the
@@ -1449,10 +1447,6 @@ DEFTREECODE (VEC_PACK_FLOAT_EXPR, "vec_pack_float_expr", tcc_binary, 2)
*/
DEFTREECODE (VEC_WIDEN_LSHIFT_HI_EXPR, "widen_lshift_hi_expr", tcc_binary, 2)
DEFTREECODE (VEC_WIDEN_LSHIFT_LO_EXPR, "widen_lshift_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_HI_EXPR, "widen_plus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_LO_EXPR, "widen_plus_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_HI_EXPR, "widen_minus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_LO_EXPR, "widen_minus_lo_expr", tcc_binary, 2)
/* PREDICT_EXPR. Specify hint for branch prediction. The
PREDICT_EXPR_PREDICTOR specify predictor and PREDICT_EXPR_OUTCOME the
--
2.17.1
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-05-31 10:07 ` Joel Hutton
@ 2022-05-31 16:46 ` Tamar Christina
2022-06-01 10:11 ` Richard Biener
1 sibling, 0 replies; 53+ messages in thread
From: Tamar Christina @ 2022-05-31 16:46 UTC (permalink / raw)
To: Joel Hutton, Richard Biener; +Cc: Richard Sandiford, gcc-patches
> Just checking there is still interest in this
Definitely, I am waiting for this to be able to send a new patch upstream 😊
Cheers,
Tamar.
> -----Original Message-----
> From: Gcc-patches <gcc-patches-
> bounces+tamar.christina=arm.com@gcc.gnu.org> On Behalf Of Joel Hutton
> via Gcc-patches
> Sent: Tuesday, May 31, 2022 11:08 AM
> To: Richard Biener <rguenther@suse.de>
> Cc: Richard Sandiford <Richard.Sandiford@arm.com>; gcc-
> patches@gcc.gnu.org
> Subject: RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as
> internal_fns
>
> > Can you post an updated patch (after the .cc renaming, and code_helper
> > now already moved to tree.h).
> >
> > Thanks,
> > Richard.
>
> Patches attached. They already incorporated the .cc rename, now rebased to
> be after the change to tree.h
>
> Joel
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-05-31 10:07 ` Joel Hutton
2022-05-31 16:46 ` Tamar Christina
@ 2022-06-01 10:11 ` Richard Biener
2022-06-06 17:20 ` Joel Hutton
1 sibling, 1 reply; 53+ messages in thread
From: Richard Biener @ 2022-06-01 10:11 UTC (permalink / raw)
To: Joel Hutton; +Cc: Richard Sandiford, gcc-patches
On Tue, 31 May 2022, Joel Hutton wrote:
> > Can you post an updated patch (after the .cc renaming, and code_helper
> > now already moved to tree.h).
> >
> > Thanks,
> > Richard.
>
> Patches attached. They already incorporated the .cc rename, now rebased to be after the change to tree.h
@@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
2, oprnd, half_type, unprom, vectype);
tree var = vect_recog_temp_ssa_var (itype, NULL);
- gimple *pattern_stmt = gimple_build_assign (var, wide_code,
- oprnd[0], oprnd[1]);
+ gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0],
oprnd[1]);
you should be able to do without the new gimple_build overload
by using
gimple_seq stmts = NULL;
gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
because 'gimple_build' is an existing API.
- if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
+ if (gimple_get_lhs (stmt) == NULL_TREE ||
+ TREE_CODE(gimple_get_lhs (stmt)) != SSA_NAME)
return false;
|| go to the next line, space after TREE_CODE
+ bool widen_arith = false;
+ gimple_match_op res_op;
+ if (!gimple_extract_op (stmt, &res_op))
+ return false;
+ code = res_op.code;
+ op_type = res_op.num_ops;
+
+ if (is_gimple_assign (stmt))
+ {
+ widen_arith = (code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR
+ || code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR);
+ }
+ else
+ widen_arith = gimple_call_flags (stmt) & ECF_WIDEN;
there seem to be formatting issues. Also shouldn't you check
if (res_op.code.is_tree_code ()) instead if is_gimple_assign?
I also don't like the ECF_WIDEN "trick", just do as with the
tree codes and explicitely enumerate widening ifns here.
gimple_extract_op is a bit heavy-weight as well, so maybe
instead simply do
if (is_gimple_assign (stmt))
{
code = gimple_assign_rhs_code (stmt);
...
}
else if (gimple_call_internal_p (stmt))
{
code = gimple_call_internal_fn (stmt);
...
}
else
return false;
+ code_helper c1=MAX_TREE_CODES, c2=MAX_TREE_CODES;
spaces before/after '='
@@ -12061,13 +12105,16 @@ supportable_widening_operation (vec_info *vinfo,
if (BYTES_BIG_ENDIAN && c1 != VEC_WIDEN_MULT_EVEN_EXPR)
std::swap (c1, c2);
+
if (code == FIX_TRUNC_EXPR)
{
unnecessary whitespace change.
diff --git a/gcc/tree.h b/gcc/tree.h
index
f84958933d51144bb6ce7cc41eca5f7f06814550..e51e34c051d9b91d1c02a4b2fefdb2b15606a36f
100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -92,6 +92,10 @@ public:
bool is_fn_code () const { return rep < 0; }
bool is_internal_fn () const;
bool is_builtin_fn () const;
+ enum tree_code as_tree_code () const { return is_tree_code () ?
+ (tree_code)* this : MAX_TREE_CODES; }
+ combined_fn as_fn_code () const { return is_fn_code () ? (combined_fn)
*this
+ : CFN_LAST;}
hmm, the other as_* functions we have are not member functions.
Also this semantically differs from the tree_code () conversion
operator (that one was supposed to be "cheap"). The existing
as_internal_fn for example is documented as being valid only if
the code is actually an internal fn. I see you are introducing
the new function as convenience to get a "safe" not-a-X value,
so maybe they should be called safe_as_tree_code () instead?
int get_rep () const { return rep; }
bool operator== (const code_helper &other) { return rep == other.rep; }
bool operator!= (const code_helper &other) { return rep != other.rep; }
@@ -6657,6 +6661,54 @@ extern unsigned fndecl_dealloc_argno (tree);
if nonnull, set the second argument to the referenced enclosing
object or pointer. Otherwise return null. */
extern tree get_attr_nonstring_decl (tree, tree * = NULL);
+/* Helper to transparently allow tree codes and builtin function codes
+ exist in one storage entity. */
+class code_helper
+{
duplicate add of code_helper.
Sorry to raise these issues so late.
Richard.
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-01 10:11 ` Richard Biener
@ 2022-06-06 17:20 ` Joel Hutton
2022-06-07 8:18 ` Richard Sandiford
0 siblings, 1 reply; 53+ messages in thread
From: Joel Hutton @ 2022-06-06 17:20 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 5524 bytes --]
> > Patches attached. They already incorporated the .cc rename, now
> > rebased to be after the change to tree.h
>
> @@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
> 2, oprnd, half_type, unprom, vectype);
>
> tree var = vect_recog_temp_ssa_var (itype, NULL);
> - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
> - oprnd[0], oprnd[1]);
> + gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0],
> oprnd[1]);
>
>
> you should be able to do without the new gimple_build overload
> by using
>
> gimple_seq stmts = NULL;
> gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
> gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
>
> because 'gimple_build' is an existing API.
Done.
The gimple_build overload was at the request of Richard Sandiford, I assume removing it is ok with you Richard S?
From Richard Sandiford:
> For example, I think we should hide this inside a new:
>
> gimple_build (var, wide_code, oprnd[0], oprnd[1]);
>
> that works directly on code_helper, similarly to the new code_helper
> gimple_build interfaces.
>
>
> - if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
> + if (gimple_get_lhs (stmt) == NULL_TREE ||
> + TREE_CODE(gimple_get_lhs (stmt)) != SSA_NAME)
> return false;
>
> || go to the next line, space after TREE_CODE
>
Done.
> + bool widen_arith = false;
> + gimple_match_op res_op;
> + if (!gimple_extract_op (stmt, &res_op))
> + return false;
> + code = res_op.code;
> + op_type = res_op.num_ops;
> +
> + if (is_gimple_assign (stmt))
> + {
> + widen_arith = (code == WIDEN_PLUS_EXPR
> + || code == WIDEN_MINUS_EXPR
> + || code == WIDEN_MULT_EXPR
> + || code == WIDEN_LSHIFT_EXPR);
> + }
> + else
> + widen_arith = gimple_call_flags (stmt) & ECF_WIDEN;
>
> there seem to be formatting issues. Also shouldn't you check
> if (res_op.code.is_tree_code ()) instead if is_gimple_assign?
> I also don't like the ECF_WIDEN "trick", just do as with the
> tree codes and explicitely enumerate widening ifns here.
>
Done. I've set widen_arith to False for the first patch as the second patch introduces the widening ifns.
> gimple_extract_op is a bit heavy-weight as well, so maybe
> instead simply do
>
> if (is_gimple_assign (stmt))
> {
> code = gimple_assign_rhs_code (stmt);
> ...
> }
> else if (gimple_call_internal_p (stmt))
> {
> code = gimple_call_internal_fn (stmt);
> ...
> }
> else
> return false;
The patch was originally written as above, it was changed to use gimple_extract_op at the request of Richard Sandiford. I prefer gimple_extract_op as it's more compact, but I don't have strong feelings. If the Richards can agree on either version I'm happy.
From Richard Sandiford:
> > + if (is_gimple_assign (stmt))
> > + {
> > + code_or_ifn = gimple_assign_rhs_code (stmt); } else
> > + code_or_ifn = gimple_call_combined_fn (stmt);
>
> It might be possible to use gimple_extract_op here (only recently added).
> This would also provide the number of operands directly, instead of
> needing "op_type". It would also provide an array of operands.
>
> + code_helper c1=MAX_TREE_CODES, c2=MAX_TREE_CODES;
>
> spaces before/after '='
>
Done.
> @@ -12061,13 +12105,16 @@ supportable_widening_operation (vec_info
> *vinfo,
> if (BYTES_BIG_ENDIAN && c1 != VEC_WIDEN_MULT_EVEN_EXPR)
> std::swap (c1, c2);
>
> +
> if (code == FIX_TRUNC_EXPR)
> {
>
> unnecessary whitespace change.
>
Fixed.
> diff --git a/gcc/tree.h b/gcc/tree.h
> index
> f84958933d51144bb6ce7cc41eca5f7f06814550..e51e34c051d9b91d1c02a4b2
> fefdb2b15606a36f
> 100644
> --- a/gcc/tree.h
> +++ b/gcc/tree.h
> @@ -92,6 +92,10 @@ public:
> bool is_fn_code () const { return rep < 0; }
> bool is_internal_fn () const;
> bool is_builtin_fn () const;
> + enum tree_code as_tree_code () const { return is_tree_code () ?
> + (tree_code)* this : MAX_TREE_CODES; }
> + combined_fn as_fn_code () const { return is_fn_code () ? (combined_fn)
> *this
> + : CFN_LAST;}
>
> hmm, the other as_* functions we have are not member functions.
> Also this semantically differs from the tree_code () conversion
> operator (that one was supposed to be "cheap"). The existing
> as_internal_fn for example is documented as being valid only if
> the code is actually an internal fn. I see you are introducing
> the new function as convenience to get a "safe" not-a-X value,
> so maybe they should be called safe_as_tree_code () instead?
>
SGTM. Done
>
> int get_rep () const { return rep; }
> bool operator== (const code_helper &other) { return rep == other.rep; }
> bool operator!= (const code_helper &other) { return rep != other.rep; }
> @@ -6657,6 +6661,54 @@ extern unsigned fndecl_dealloc_argno (tree);
> if nonnull, set the second argument to the referenced enclosing
> object or pointer. Otherwise return null. */
> extern tree get_attr_nonstring_decl (tree, tree * = NULL);
> +/* Helper to transparently allow tree codes and builtin function codes
> + exist in one storage entity. */
> +class code_helper
> +{
>
> duplicate add of code_helper.
Fixed.
Tests are being re-run.
Ok, with changes?
[-- Attachment #2: 0001-Refactor-to-allow-internal_fn-s.patch --]
[-- Type: application/octet-stream, Size: 23246 bytes --]
From 58d1f19224bd6501b5238916871cf2c0f3ba8bd0 Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Wed, 25 Aug 2021 14:31:15 +0100
Subject: [PATCH 1/3] Refactor to allow internal_fn's
Hi all,
This refactor allows widening patterns (such as widen_plus/widen_minus) to be represented as
either internal_fns or tree_codes.
[vect-patterns] Refactor as internal_fn's
Refactor vect-patterns to allow patterns to be internal_fns starting
with widening_plus/minus patterns
gcc/ChangeLog:
* gimple-match.h (class code_helper): Add safe_as_internal_fn, safe_as_tree_code
helper functions.
* tree-core.h (ECF_WIDEN): Flag to mark internal_fn as widening.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Refactor to
use code_helper.
* tree-vect-stmts.cc (vect_gen_widened_results_half): Refactor to
use code_helper.
(vect_create_vectorized_promotion_stmts): Refactor to use
code_helper.
(vectorizable_conversion): Refactor to use code_helper.
gimple_call or gimple_assign.
(supportable_widening_operation): Refactor to use code_helper.
(supportable_narrowing_operation): Refactor to use code_helper.
* tree-vectorizer.h (supportable_widening_operation): Change
prototype to use code_helper.
(supportable_narrowing_operation): change prototype to use
code_helper.
---
gcc/tree-core.h | 3 +
gcc/tree-vect-patterns.cc | 11 +-
gcc/tree-vect-stmts.cc | 217 +++++++++++++++++++++++---------------
gcc/tree-vectorizer.h | 11 +-
gcc/tree.h | 4 +
5 files changed, 153 insertions(+), 93 deletions(-)
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index ab5fa01e5cb5fb56c1964b93b014ed55a4aa704a..cff6211080bced0bffb39e98039a6550897acf77 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -96,6 +96,9 @@ struct die_struct;
/* Nonzero if this is a cold function. */
#define ECF_COLD (1 << 15)
+/* Nonzero if this is a widening function. */
+#define ECF_WIDEN (1 << 16)
+
/* Call argument flags. */
/* Nonzero if the argument is not used by the function. */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 0fad4dbd0945c6c176f3457b751e812f17fcd148..c011b8ede3c266b59f731e316efbec7d98e91068 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -25,6 +25,8 @@ along with GCC; see the file COPYING3. If not see
#include "rtl.h"
#include "tree.h"
#include "gimple.h"
+#include "gimple-iterator.h"
+#include "gimple-fold.h"
#include "ssa.h"
#include "expmed.h"
#include "optabs-tree.h"
@@ -1348,7 +1350,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
- tree_code orig_code, tree_code wide_code,
+ tree_code orig_code, code_helper wide_code,
bool shift_p, const char *name)
{
gimple *last_stmt = last_stmt_info->stmt;
@@ -1391,7 +1393,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
vecctype = get_vectype_for_scalar_type (vinfo, ctype);
}
- enum tree_code dummy_code;
+ code_helper dummy_code;
int dummy_int;
auto_vec<tree> dummy_vec;
if (!vectype
@@ -1412,8 +1414,9 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
2, oprnd, half_type, unprom, vectype);
tree var = vect_recog_temp_ssa_var (itype, NULL);
- gimple *pattern_stmt = gimple_build_assign (var, wide_code,
- oprnd[0], oprnd[1]);
+ gimple_seq stmts = NULL;
+ gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
+ gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
if (vecctype != vecitype)
pattern_stmt = vect_convert_output (vinfo, last_stmt_info, ctype,
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 346d8ce280437e00bfeb19a4b4adc59eb96207f9..9b31425352689d409b8c0aa0c1d5c69e72db869a 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4636,7 +4636,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
STMT_INFO is the original scalar stmt that we are vectorizing. */
static gimple *
-vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
+vect_gen_widened_results_half (vec_info *vinfo, code_helper ch,
tree vec_oprnd0, tree vec_oprnd1, int op_type,
tree vec_dest, gimple_stmt_iterator *gsi,
stmt_vec_info stmt_info)
@@ -4645,14 +4645,15 @@ vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
tree new_temp;
/* Generate half of the widened result: */
- gcc_assert (op_type == TREE_CODE_LENGTH (code));
if (op_type != binary_op)
vec_oprnd1 = NULL;
- new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0, vec_oprnd1);
+
+ gimple_seq stmts = NULL;
+ gimple_build (&stmts, ch, vec_oprnd0, vec_oprnd1);
+ new_stmt = gimple_seq_last_stmt (stmts);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
-
return new_stmt;
}
@@ -4729,8 +4730,8 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
- enum tree_code code2, int op_type)
+ code_helper ch1,
+ code_helper ch2, int op_type)
{
int i;
tree vop0, vop1, new_tmp1, new_tmp2;
@@ -4746,10 +4747,10 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vop1 = NULL_TREE;
/* Generate the two halves of promotion operation. */
- new_stmt1 = vect_gen_widened_results_half (vinfo, code1, vop0, vop1,
+ new_stmt1 = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
- new_stmt2 = vect_gen_widened_results_half (vinfo, code2, vop0, vop1,
+ new_stmt2 = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
if (is_gimple_call (new_stmt1))
@@ -4846,8 +4847,9 @@ vectorizable_conversion (vec_info *vinfo,
tree scalar_dest;
tree op0, op1 = NULL_TREE;
loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
- enum tree_code code, code1 = ERROR_MARK, code2 = ERROR_MARK;
- enum tree_code codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
+ tree_code tc1;
+ code_helper code, code1, code2;
+ code_helper codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
tree new_temp;
enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type};
int ndts = 2;
@@ -4876,31 +4878,42 @@ vectorizable_conversion (vec_info *vinfo,
&& ! vec_stmt)
return false;
- gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
- if (!stmt)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return false;
- if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
+ if (gimple_get_lhs (stmt) == NULL_TREE
+ || TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- code = gimple_assign_rhs_code (stmt);
- if (!CONVERT_EXPR_CODE_P (code)
- && code != FIX_TRUNC_EXPR
- && code != FLOAT_EXPR
- && code != WIDEN_PLUS_EXPR
- && code != WIDEN_MINUS_EXPR
- && code != WIDEN_MULT_EXPR
- && code != WIDEN_LSHIFT_EXPR)
+ if (TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
- op_type = TREE_CODE_LENGTH (code);
+ bool widen_arith = false;
+ gimple_match_op res_op;
+ if (!gimple_extract_op (stmt, &res_op))
+ return false;
+ code = res_op.code;
+ op_type = res_op.num_ops;
+
+ if (res_op.code.is_tree_code ())
+ {
+ widen_arith = (code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR
+ || code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR);
+ }
+ else
+ widen_arith = false;
+
+ if (!widen_arith
+ && !CONVERT_EXPR_CODE_P (code)
+ && code != FIX_TRUNC_EXPR
+ && code != FLOAT_EXPR)
+ return false;
/* Check types of lhs and rhs. */
- scalar_dest = gimple_assign_lhs (stmt);
+ scalar_dest = gimple_get_lhs (stmt);
lhs_type = TREE_TYPE (scalar_dest);
vectype_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -4938,10 +4951,14 @@ vectorizable_conversion (vec_info *vinfo,
if (op_type == binary_op)
{
- gcc_assert (code == WIDEN_MULT_EXPR || code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR || code == WIDEN_MINUS_EXPR);
+ gcc_assert (code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR
+ || code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR);
- op1 = gimple_assign_rhs2 (stmt);
+
+ op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
+ gimple_call_arg (stmt, 0);
tree vectype1_in;
if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
&op1, &slp_op1, &dt[1], &vectype1_in))
@@ -5025,8 +5042,12 @@ vectorizable_conversion (vec_info *vinfo,
&& code != FLOAT_EXPR
&& !CONVERT_EXPR_CODE_P (code))
return false;
- if (supportable_convert_operation (code, vectype_out, vectype_in, &code1))
+ if (supportable_convert_operation (code.safe_as_tree_code (), vectype_out,
+ vectype_in, &tc1))
+ {
+ code1 = tc1;
break;
+ }
/* FALLTHRU */
unsupported:
if (dump_enabled_p ())
@@ -5037,9 +5058,11 @@ vectorizable_conversion (vec_info *vinfo,
case WIDEN:
if (known_eq (nunits_in, nunits_out))
{
- if (!supportable_half_widening_operation (code, vectype_out,
- vectype_in, &code1))
+ if (!supportable_half_widening_operation (code.safe_as_tree_code (),
+ vectype_out, vectype_in,
+ &tc1))
goto unsupported;
+ code1 = tc1;
gcc_assert (!(multi_step_cvt && op_type == binary_op));
break;
}
@@ -5073,14 +5096,17 @@ vectorizable_conversion (vec_info *vinfo,
if (GET_MODE_SIZE (rhs_mode) == fltsz)
{
- if (!supportable_convert_operation (code, vectype_out,
- cvt_type, &codecvt1))
+ tc1 = ERROR_MARK;
+ if (!supportable_convert_operation (code.safe_as_tree_code (),
+ vectype_out,
+ cvt_type, &tc1))
goto unsupported;
+ codecvt1 = tc1;
}
- else if (!supportable_widening_operation (vinfo, code, stmt_info,
- vectype_out, cvt_type,
- &codecvt1, &codecvt2,
- &multi_step_cvt,
+ else if (!supportable_widening_operation (vinfo, code,
+ stmt_info, vectype_out,
+ cvt_type, &codecvt1,
+ &codecvt2, &multi_step_cvt,
&interm_types))
continue;
else
@@ -5088,8 +5114,9 @@ vectorizable_conversion (vec_info *vinfo,
if (supportable_widening_operation (vinfo, NOP_EXPR, stmt_info,
cvt_type,
- vectype_in, &code1, &code2,
- &multi_step_cvt, &interm_types))
+ vectype_in, &code1,
+ &code2, &multi_step_cvt,
+ &interm_types))
{
found_mode = true;
break;
@@ -5111,10 +5138,14 @@ vectorizable_conversion (vec_info *vinfo,
case NARROW:
gcc_assert (op_type == unary_op);
- if (supportable_narrowing_operation (code, vectype_out, vectype_in,
- &code1, &multi_step_cvt,
+ if (supportable_narrowing_operation (code.safe_as_tree_code (), vectype_out,
+ vectype_in,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
if (code != FIX_TRUNC_EXPR
|| GET_MODE_SIZE (lhs_mode) >= GET_MODE_SIZE (rhs_mode))
@@ -5125,13 +5156,18 @@ vectorizable_conversion (vec_info *vinfo,
cvt_type = get_same_sized_vectype (cvt_type, vectype_in);
if (cvt_type == NULL_TREE)
goto unsupported;
- if (!supportable_convert_operation (code, cvt_type, vectype_in,
- &codecvt1))
+ if (!supportable_convert_operation (code.safe_as_tree_code (), cvt_type,
+ vectype_in,
+ &tc1))
goto unsupported;
+ codecvt1 = tc1;
if (supportable_narrowing_operation (NOP_EXPR, vectype_out, cvt_type,
- &code1, &multi_step_cvt,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
goto unsupported;
default:
@@ -5245,8 +5281,9 @@ vectorizable_conversion (vec_info *vinfo,
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
/* Arguments are ready, create the new vector stmt. */
- gcc_assert (TREE_CODE_LENGTH (code1) == unary_op);
- gassign *new_stmt = gimple_build_assign (vec_dest, code1, vop0);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) code1) == unary_op);
+ gassign *new_stmt = gimple_build_assign (vec_dest,
+ code1.safe_as_tree_code (), vop0);
new_temp = make_ssa_name (vec_dest, new_stmt);
gimple_assign_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
@@ -5278,7 +5315,7 @@ vectorizable_conversion (vec_info *vinfo,
for (i = multi_step_cvt; i >= 0; i--)
{
tree this_dest = vec_dsts[i];
- enum tree_code c1 = code1, c2 = code2;
+ code_helper c1 = code1, c2 = code2;
if (i == 0 && codecvt2 != ERROR_MARK)
{
c1 = codecvt1;
@@ -5288,7 +5325,8 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_half_widening_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
this_dest, gsi,
- c1, op_type);
+ c1.safe_as_tree_code (),
+ op_type);
else
vect_create_vectorized_promotion_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
@@ -5301,9 +5339,11 @@ vectorizable_conversion (vec_info *vinfo,
gimple *new_stmt;
if (cvt_type)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, codecvt1, vop0);
+ new_stmt = gimple_build_assign (new_temp,
+ codecvt1.safe_as_tree_code (),
+ vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -5327,10 +5367,10 @@ vectorizable_conversion (vec_info *vinfo,
if (cvt_type)
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH (((tree_code) codecvt1)) == unary_op);
new_temp = make_ssa_name (vec_dest);
gassign *new_stmt
- = gimple_build_assign (new_temp, codecvt1, vop0);
+ = gimple_build_assign (new_temp, codecvt1.safe_as_tree_code (), vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
vec_oprnds0[i] = new_temp;
}
@@ -5338,7 +5378,7 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_vectorized_demotion_stmts (vinfo, &vec_oprnds0,
multi_step_cvt,
stmt_info, vec_dsts, gsi,
- slp_node, code1);
+ slp_node, code1.safe_as_tree_code ());
break;
}
if (!slp_node)
@@ -11926,9 +11966,11 @@ vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype)
bool
supportable_widening_operation (vec_info *vinfo,
- enum tree_code code, stmt_vec_info stmt_info,
+ code_helper code,
+ stmt_vec_info stmt_info,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, enum tree_code *code2,
+ code_helper *code1,
+ code_helper *code2,
int *multi_step_cvt,
vec<tree> *interm_types)
{
@@ -11939,7 +11981,7 @@ supportable_widening_operation (vec_info *vinfo,
optab optab1, optab2;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
- enum tree_code c1, c2;
+ code_helper c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
int i;
tree prev_type, intermediate_type;
machine_mode intermediate_mode, prev_mode;
@@ -11949,7 +11991,7 @@ supportable_widening_operation (vec_info *vinfo,
if (loop_info)
vect_loop = LOOP_VINFO_LOOP (loop_info);
- switch (code)
+ switch (code.safe_as_tree_code ())
{
case WIDEN_MULT_EXPR:
/* The result of a vectorized widening operation usually requires
@@ -11990,8 +12032,9 @@ supportable_widening_operation (vec_info *vinfo,
&& !nested_in_vect_loop_p (vect_loop, stmt_info)
&& supportable_widening_operation (vinfo, VEC_WIDEN_MULT_EVEN_EXPR,
stmt_info, vectype_out,
- vectype_in, code1, code2,
- multi_step_cvt, interm_types))
+ vectype_in, code1,
+ code2, multi_step_cvt,
+ interm_types))
{
/* Elements in a vector with vect_used_by_reduction property cannot
be reordered if the use chain with this property does not have the
@@ -12054,6 +12097,9 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_UNPACK_FIX_TRUNC_HI_EXPR;
break;
+ case MAX_TREE_CODES:
+ break;
+
default:
gcc_unreachable ();
}
@@ -12064,10 +12110,12 @@ supportable_widening_operation (vec_info *vinfo,
if (code == FIX_TRUNC_EXPR)
{
/* The signedness is determined from output operand. */
- optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
- optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype_out,
+ optab_default);
}
- else if (CONVERT_EXPR_CODE_P (code)
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
&& VECTOR_BOOLEAN_TYPE_P (wide_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
&& TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
@@ -12080,8 +12128,8 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype, optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype, optab_default);
}
if (!optab1 || !optab2)
@@ -12092,8 +12140,12 @@ supportable_widening_operation (vec_info *vinfo,
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- *code1 = c1;
- *code2 = c2;
+ if (code.is_tree_code ())
+ {
+ *code1 = c1;
+ *code2 = c2;
+ }
+
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
@@ -12114,7 +12166,7 @@ supportable_widening_operation (vec_info *vinfo,
prev_type = vectype;
prev_mode = vec_mode;
- if (!CONVERT_EXPR_CODE_P (code))
+ if (!CONVERT_EXPR_CODE_P ((tree_code) code))
return false;
/* We assume here that there will not be more than MAX_INTERM_CVT_STEPS
@@ -12145,8 +12197,10 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab3 = optab_for_tree_code (c1, intermediate_type, optab_default);
- optab4 = optab_for_tree_code (c2, intermediate_type, optab_default);
+ optab3 = optab_for_tree_code (c1.safe_as_tree_code (), intermediate_type,
+ optab_default);
+ optab4 = optab_for_tree_code (c2.safe_as_tree_code (), intermediate_type,
+ optab_default);
}
if (!optab3 || !optab4
@@ -12181,7 +12235,6 @@ supportable_widening_operation (vec_info *vinfo,
return false;
}
-
/* Function supportable_narrowing_operation
Check whether an operation represented by the code CODE is a
@@ -12205,7 +12258,7 @@ supportable_widening_operation (vec_info *vinfo,
bool
supportable_narrowing_operation (enum tree_code code,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, int *multi_step_cvt,
+ tree_code* _code1, int *multi_step_cvt,
vec<tree> *interm_types)
{
machine_mode vec_mode;
@@ -12217,8 +12270,8 @@ supportable_narrowing_operation (enum tree_code code,
tree intermediate_type, prev_type;
machine_mode intermediate_mode, prev_mode;
int i;
- unsigned HOST_WIDE_INT n_elts;
bool uns;
+ tree_code * code1 = (tree_code*) _code1;
*multi_step_cvt = 0;
switch (code)
@@ -12227,9 +12280,8 @@ supportable_narrowing_operation (enum tree_code code,
c1 = VEC_PACK_TRUNC_EXPR;
if (VECTOR_BOOLEAN_TYPE_P (narrow_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
- && SCALAR_INT_MODE_P (TYPE_MODE (vectype))
- && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&n_elts)
- && n_elts < BITS_PER_UNIT)
+ && TYPE_MODE (narrow_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
optab1 = vec_pack_sbool_trunc_optab;
else
optab1 = optab_for_tree_code (c1, vectype, optab_default);
@@ -12320,9 +12372,8 @@ supportable_narrowing_operation (enum tree_code code,
= lang_hooks.types.type_for_mode (intermediate_mode, uns);
if (VECTOR_BOOLEAN_TYPE_P (intermediate_type)
&& VECTOR_BOOLEAN_TYPE_P (prev_type)
- && SCALAR_INT_MODE_P (prev_mode)
- && TYPE_VECTOR_SUBPARTS (intermediate_type).is_constant (&n_elts)
- && n_elts < BITS_PER_UNIT)
+ && intermediate_mode == prev_mode
+ && SCALAR_INT_MODE_P (prev_mode))
interm_optab = vec_pack_sbool_trunc_optab;
else
interm_optab
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 642eb0aeb21264cd736a479b1ec25357abef29cd..50ff8eeac1e6b9859302bd784f10ee3d8ff4b4dc 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2120,13 +2120,12 @@ extern bool vect_is_simple_use (vec_info *, stmt_vec_info, slp_tree,
enum vect_def_type *,
tree *, stmt_vec_info * = NULL);
extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
-extern bool supportable_widening_operation (vec_info *,
- enum tree_code, stmt_vec_info,
- tree, tree, enum tree_code *,
- enum tree_code *, int *,
- vec<tree> *);
+extern bool supportable_widening_operation (vec_info*, code_helper,
+ stmt_vec_info, tree, tree,
+ code_helper*, code_helper*,
+ int*, vec<tree> *);
extern bool supportable_narrowing_operation (enum tree_code, tree, tree,
- enum tree_code *, int *,
+ tree_code *, int *,
vec<tree> *);
extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
diff --git a/gcc/tree.h b/gcc/tree.h
index f84958933d51144bb6ce7cc41eca5f7f06814550..00b0e4d1c696633fe38baad5295b1f90398d53fc 100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -92,6 +92,10 @@ public:
bool is_fn_code () const { return rep < 0; }
bool is_internal_fn () const;
bool is_builtin_fn () const;
+ enum tree_code safe_as_tree_code () const { return is_tree_code () ?
+ (tree_code)* this : MAX_TREE_CODES; }
+ combined_fn safe_as_fn_code () const { return is_fn_code () ? (combined_fn) *this
+ : CFN_LAST;}
int get_rep () const { return rep; }
bool operator== (const code_helper &other) { return rep == other.rep; }
bool operator!= (const code_helper &other) { return rep != other.rep; }
--
2.17.1
[-- Attachment #3: 0002-Refactor-widen_plus-as-internal_fn.patch --]
[-- Type: application/octet-stream, Size: 23132 bytes --]
From 233a24f2a4eeced2fd4e99578e6ea81ec8622192 Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Wed, 26 Jan 2022 14:00:17 +0000
Subject: [PATCH 2/3] Refactor widen_plus as internal_fn
This patch replaces the existing tree_code widen_plus and widen_minus
patterns with internal_fn versions.
DEF_INTERNAL_OPTAB_MULTI_FN is like DEF_INTERNAL_OPTAB_FN except it provides convenience wrappers for defining conversions that require a hi/lo split, like widening and narrowing operations. Each definition for <NAME> will require an optab named <OPTAB> and two other optabs that you specify for signed and unsigned. The hi/lo pair is necessary because the widening operations take n narrow elements as inputs and return n/2 wide elements as outputs. The 'lo' operation operates on the first n/2 elements of input. The 'hi' operation operates on the second n/2 elements of input. Defining an internal_fn along with hi/lo variations allows a single internal function to be returned from a vect_recog function that will later be expanded to hi/lo.
DEF_INTERNAL_OPTAB_MULTI_FN is used in internal-fn.def to register a widening internal_fn. It is defined differently in different places and internal-fn.def is sourced from those places so the parameters given can be reused.
internal-fn.c: defined to expand to hi/lo signed/unsigned optabs, later defined to generate the 'expand_' functions for the hi/lo versions of the fn.
internal-fn.def: defined to invoke DEF_INTERNAL_OPTAB_FN for the original and hi/lo variants of the internal_fn
For example:
IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>addl_hi_<mode> -> (u/s)addl2
IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>addl_lo_<mode> -> (u/s)addl
This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
gcc/ChangeLog:
2022-04-13 Joel Hutton <joel.hutton@arm.com>
2022-04-13 Tamar Christina <tamar.christina@arm.com>
* internal-fn.cc (INCLUDE_MAP): Include maps for use in optab
lookup.
(DEF_INTERNAL_OPTAB_MULTI_FN): Macro to define an internal_fn that
expands into multiple internal_fns (for widening).
(ifn_cmp): Function to compare ifn's for sorting/searching.
(lookup_multi_ifn_optab): Add lookup function.
(lookup_multi_internal_fn): Add lookup function.
(commutative_binary_fn_p): Add widen_plus fn's.
* internal-fn.def (DEF_INTERNAL_OPTAB_MULTI_FN): Define widening
plus,minus functions.
(VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS tree code.
(VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS tree code.
* internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
(lookup_multi_ifn_optab): Add prototype.
(lookup_multi_internal_fn): Add prototype.
* optabs.cc (commutative_optab_p): Add widening plus, minus optabs.
* optabs.def (OPTAB_CD): widen add, sub optabs
* tree-core.h (ECF_MULTI): Flag to indicate if a function decays
into hi/lo parts.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
patterns with a hi/lo split.
(vect_recog_widen_plus_pattern): Refactor to return
IFN_VECT_WIDEN_PLUS.
(vect_recog_widen_minus_pattern): Refactor to return new
IFN_VEC_WIDEN_MINUS.
* tree-vect-stmts.cc (vectorizable_conversion): Add widen plus/minus
ifn
support.
(supportable_widening_operation): Add widen plus/minus ifn support.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-widen-add.c: Test that new
IFN_VEC_WIDEN_PLUS is being used.
* gcc.target/aarch64/vect-widen-sub.c: Test that new
IFN_VEC_WIDEN_MINUS is being used.
---
gcc/internal-fn.cc | 107 ++++++++++++++++++
gcc/internal-fn.def | 23 ++++
gcc/internal-fn.h | 7 ++
gcc/optabs.cc | 12 +-
gcc/optabs.def | 2 +
.../gcc.target/aarch64/vect-widen-add.c | 4 +-
.../gcc.target/aarch64/vect-widen-sub.c | 4 +-
gcc/tree-core.h | 4 +
gcc/tree-vect-patterns.cc | 37 ++++--
gcc/tree-vect-stmts.cc | 65 ++++++++++-
10 files changed, 248 insertions(+), 17 deletions(-)
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 8b1733e20c4455e4e8c383c92fe859f4256cae69..e95b13af884f67990ad43c286990a351e2bd641b 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -17,6 +17,7 @@ You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3. If not see
<http://www.gnu.org/licenses/>. */
+#define INCLUDE_MAP
#include "config.h"
#include "system.h"
#include "coretypes.h"
@@ -70,6 +71,26 @@ const int internal_fn_flags_array[] = {
0
};
+const enum internal_fn internal_fn_hilo_keys_array[] = {
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ IFN_##NAME##_LO, \
+ IFN_##NAME##_HI,
+#include "internal-fn.def"
+ IFN_LAST
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+};
+
+const optab internal_fn_hilo_values_array[] = {
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ SOPTAB##_lo_optab, UOPTAB##_lo_optab, \
+ SOPTAB##_hi_optab, UOPTAB##_hi_optab,
+#include "internal-fn.def"
+ unknown_optab, unknown_optab
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+};
+
/* Return the internal function called NAME, or IFN_LAST if there's
no such function. */
@@ -90,6 +111,62 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+static int
+ifn_cmp (const void *a_, const void *b_)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ auto *a = (const std::pair<ifn_pair, optab> *)a_;
+ auto *b = (const std::pair<ifn_pair, optab> *)b_;
+ return (int) (a->first.first) - (b->first.first);
+}
+
+/* Return the optab belonging to the given internal function NAME for the given
+ SIGN or unknown_optab. */
+
+optab
+lookup_multi_ifn_optab (enum internal_fn fn, unsigned sign)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ typedef auto_vec <std::pair<ifn_pair, optab>>fn_to_optab_map_type;
+ static fn_to_optab_map_type *fn_to_optab_map;
+
+ if (!fn_to_optab_map)
+ {
+ unsigned num
+ = sizeof (internal_fn_hilo_keys_array) / sizeof (enum internal_fn);
+ fn_to_optab_map = new fn_to_optab_map_type ();
+ for (unsigned int i = 0; i < num - 1; ++i)
+ {
+ enum internal_fn fn = internal_fn_hilo_keys_array[i];
+ optab v1 = internal_fn_hilo_values_array[2*i];
+ optab v2 = internal_fn_hilo_values_array[2*i + 1];
+ ifn_pair key1 (fn, 0);
+ fn_to_optab_map->safe_push ({key1, v1});
+ ifn_pair key2 (fn, 1);
+ fn_to_optab_map->safe_push ({key2, v2});
+ }
+ fn_to_optab_map->qsort(ifn_cmp);
+ }
+
+ ifn_pair new_pair (fn, sign ? 1 : 0);
+ optab tmp;
+ std::pair<ifn_pair,optab> pair_wrap (new_pair, tmp);
+ auto entry = fn_to_optab_map->bsearch (&pair_wrap, ifn_cmp);
+ return entry != fn_to_optab_map->end () ? entry->second : unknown_optab;
+}
+
+extern void
+lookup_multi_internal_fn (enum internal_fn ifn, enum internal_fn *lo,
+ enum internal_fn *hi)
+{
+ int ecf_flags = internal_fn_flags (ifn);
+ gcc_assert (ecf_flags & ECF_MULTI);
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -3906,6 +3983,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -3991,6 +4071,32 @@ set_edom_supported_p (void)
#endif
}
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(CODE, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ static void \
+ expand_##CODE (internal_fn, gcall *) \
+ { \
+ gcc_unreachable (); \
+ } \
+ static void \
+ expand_##CODE##_LO (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_lo##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_lo##_optab); \
+ } \
+ static void \
+ expand_##CODE##_HI (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_hi##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_hi##_optab); \
+ }
+
#define DEF_INTERNAL_OPTAB_FN(CODE, FLAGS, OPTAB, TYPE) \
static void \
expand_##CODE (internal_fn fn, gcall *stmt) \
@@ -4007,6 +4113,7 @@ set_edom_supported_p (void)
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index d2d550d358606022b1cb44fa842f06e0be507bc3..4635a9c8af9ad27bb05d7510388d0fe2270428e5 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -82,6 +82,13 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_OPTAB_MULTI_FN is like DEF_INTERNAL_OPTAB_FN except it
+ provides convenience wrappers for defining conversions that require a
+ hi/lo split, like widening and narrowing operations. Each definition
+ for <NAME> will require an optab named <OPTAB> and two other optabs that
+ you specify for signed and unsigned.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -120,6 +127,14 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME, FLAGS | ECF_MULTI, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, unknown, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, unknown, TYPE)
+#endif
+
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -292,6 +307,14 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_OPTAB_MULTI_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_WIDEN | ECF_NOTHROW,
+ vec_widen_add, vec_widen_saddl, vec_widen_uaddl,
+ binary)
+DEF_INTERNAL_OPTAB_MULTI_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_WIDEN | ECF_NOTHROW,
+ vec_widen_sub, vec_widen_ssubl, vec_widen_usubl,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 23c014a963c4d72da92c763db87ee486a2adb485..b35de19747d251d19dc13de1e0323368bd2ebdf2 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,9 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern optab lookup_multi_ifn_optab (enum internal_fn, unsigned);
+extern void lookup_multi_internal_fn (enum internal_fn, enum internal_fn *,
+ enum internal_fn *);
/* Return the ECF_* flags for function FN. */
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index c0a68471d2ddf08bc0e6a3fd592ebb9f05e516c1..7e904b3e154d018779bb1a36de74e6997f70e193 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,17 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_add_optab
+ || binoptab == vec_widen_sub_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_ssubl_hi_optab
+ || binoptab == vec_widen_ssubl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab
+ || binoptab == vec_widen_usubl_hi_optab
+ || binoptab == vec_widen_usubl_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 801310ebaa7d469520809bb7efed6820f8eb866b..a7881dcb49e4ef07d8f07aa31214eb3a7a944e2e 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -78,6 +78,8 @@ OPTAB_CD(smsub_widen_optab, "msub$b$a4")
OPTAB_CD(umsub_widen_optab, "umsub$b$a4")
OPTAB_CD(ssmsub_widen_optab, "ssmsub$b$a4")
OPTAB_CD(usmsub_widen_optab, "usmsub$a$b4")
+OPTAB_CD(vec_widen_add_optab, "add$a$b3")
+OPTAB_CD(vec_widen_sub_optab, "sub$a$b3")
OPTAB_CD(vec_load_lanes_optab, "vec_load_lanes$a$b")
OPTAB_CD(vec_store_lanes_optab, "vec_store_lanes$a$b")
OPTAB_CD(vec_mask_load_lanes_optab, "vec_mask_load_lanes$a$b")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index cff6211080bced0bffb39e98039a6550897acf77..d0c8b812cfb9c3ac83bf25fff0431b08cb7d823d 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -99,6 +99,10 @@ struct die_struct;
/* Nonzero if this is a widening function. */
#define ECF_WIDEN (1 << 16)
+/* Nonzero if this is a function that decomposes into a lo/hi operation. */
+#define ECF_MULTI (1 << 17)
+
+
/* Call argument flags. */
/* Nonzero if the argument is not used by the function. */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index c011b8ede3c266b59f731e316efbec7d98e91068..268f5402fcdd5ec5bfb806db8c410e701c771275 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1351,14 +1351,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1426,6 +1428,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1439,26 +1455,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_popcount_pattern
@@ -5622,6 +5642,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 9b31425352689d409b8c0aa0c1d5c69e72db869a..9af0d107fdafb959db10d87e4e0ba5fda4e47bd7 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4904,7 +4904,8 @@ vectorizable_conversion (vec_info *vinfo,
|| code == WIDEN_LSHIFT_EXPR);
}
else
- widen_arith = false;
+ widen_arith = (code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -4954,7 +4955,9 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
+ || code == WIDEN_MINUS_EXPR
+ || code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
@@ -12126,12 +12129,62 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
- {
- optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype, optab_default);
- optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype, optab_default);
+
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn (code.safe_as_fn_code ());
+ int ecf_flags = internal_fn_flags (ifn);
+ gcc_assert (ecf_flags & ECF_MULTI);
+
+ switch (code.safe_as_fn_code ())
+ {
+ case CFN_VEC_WIDEN_PLUS:
+ break;
+ case CFN_VEC_WIDEN_MINUS:
+ break;
+ case CFN_LAST:
+ default:
+ return false;
+ }
+
+ internal_fn lo, hi;
+ lookup_multi_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
}
+ if (code.is_tree_code ())
+ {
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype_out,
+ optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype,
+ optab_default);
+ }
+ }
+
if (!optab1 || !optab2)
return false;
--
2.17.1
[-- Attachment #4: 0003-Remove-widen_plus-minus_expr-tree-codes.patch --]
[-- Type: application/octet-stream, Size: 19519 bytes --]
From 061e11fa3c76c42640ab6467858e057e3067a6d3 Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Fri, 28 Jan 2022 12:04:44 +0000
Subject: [PATCH 3/3] Remove widen_plus/minus_expr tree codes
This patch removes the old widen plus/minus tree codes which have been
replaced by internal functions.
gcc/ChangeLog:
* doc/generic.texi: Remove old tree codes.
* expr.cc (expand_expr_real_2): Remove old tree code cases.
* gimple-pretty-print.cc (dump_binary_rhs): Remove old tree code
cases.
* optabs-tree.cc (optab_for_tree_code): Remove old tree code cases.
(supportable_half_widening_operation): Remove old tree code cases.
* tree-cfg.cc (verify_gimple_assign_binary): Remove old tree code
cases.
* tree-inline.cc (estimate_operator_cost): Remove old tree code
cases.
* tree-pretty-print.cc (dump_generic_node): Remove tree code definition.
(op_symbol_code): Remove old tree code
cases.
* tree-vect-data-refs.cc (vect_get_smallest_scalar_type): Remove old tree code
cases.
(vect_analyze_data_ref_accesses): Remove old tree code
cases.
* tree-vect-generic.cc (expand_vector_operations_1): Remove old tree code
cases.
* tree-vect-patterns.cc (vect_widened_op_tree): Refactor ot replace
usage in vect_recog_sad_pattern.
(vect_recog_sad_pattern): Replace tree code widening pattern with
internal function.
(vect_recog_average_pattern): Replace tree code widening pattern
with internal function.
* tree-vect-stmts.cc (vectorizable_conversion): Remove old tree code
cases.
(supportable_widening_operation): Remove old tree code
cases.
* tree.def (WIDEN_PLUS_EXPR): Remove tree code definition.
(WIDEN_MINUS_EXPR): Remove tree code definition.
(VEC_WIDEN_PLUS_HI_EXPR): Remove tree code definition.
(VEC_WIDEN_PLUS_LO_EXPR): Remove tree code definition.
(VEC_WIDEN_MINUS_HI_EXPR): Remove tree code definition.
(VEC_WIDEN_MINUS_LO_EXPR): Remove tree code definition.
---
gcc/doc/generic.texi | 31 -------------------------------
gcc/expr.cc | 6 ------
gcc/gimple-pretty-print.cc | 4 ----
gcc/optabs-tree.cc | 24 ------------------------
gcc/tree-cfg.cc | 6 ------
gcc/tree-inline.cc | 6 ------
gcc/tree-pretty-print.cc | 12 ------------
gcc/tree-vect-data-refs.cc | 8 +++-----
gcc/tree-vect-generic.cc | 4 ----
gcc/tree-vect-patterns.cc | 36 +++++++++++++++++++++++++-----------
gcc/tree-vect-stmts.cc | 18 ++----------------
gcc/tree.def | 6 ------
12 files changed, 30 insertions(+), 131 deletions(-)
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index e5f9d1be8ea81f3da002ec3bb925590d331a2551..344045efd419b0cc3a11771acf70d2fd279c48ac 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,10 +1811,6 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1861,33 +1857,6 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
-@item VEC_WIDEN_PLUS_HI_EXPR
-@itemx VEC_WIDEN_PLUS_LO_EXPR
-These nodes represent widening vector addition of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The result
-is a vector that contains half as many elements, of an integral type whose size
-is twice as wide. In the case of @code{VEC_WIDEN_PLUS_HI_EXPR} the high
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products. In the case of @code{VEC_WIDEN_PLUS_LO_EXPR} the low
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products.
-
-@item VEC_WIDEN_MINUS_HI_EXPR
-@itemx VEC_WIDEN_MINUS_LO_EXPR
-These nodes represent widening vector subtraction of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The high/low
-elements of the second vector are subtracted from the high/low elements of the
-first. The result is a vector that contains half as many elements, of an
-integral type whose size is twice as wide. In the case of
-@code{VEC_WIDEN_MINUS_HI_EXPR} the high @code{N/2} elements of the second
-vector are subtracted from the high @code{N/2} of the first to produce the
-vector of @code{N/2} products. In the case of
-@code{VEC_WIDEN_MINUS_LO_EXPR} the low @code{N/2} elements of the second
-vector are subtracted from the low @code{N/2} of the first to produce the
-vector of @code{N/2} products.
-
@item VEC_UNPACK_HI_EXPR
@itemx VEC_UNPACK_LO_EXPR
These nodes represent unpacking of the high and low parts of the input vector,
diff --git a/gcc/expr.cc b/gcc/expr.cc
index fb062dc847577ec9dc2c951330f4cfadcc869325..4e3655070400cee086c2fdc6ac5bbe08d303de5e 100644
--- a/gcc/expr.cc
+++ b/gcc/expr.cc
@@ -9371,8 +9371,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
target, unsignedp);
return target;
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_MULT_EXPR:
/* If first operand is constant, swap them.
Thus the following special case checks need only
@@ -10150,10 +10148,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
return temp;
}
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/gimple-pretty-print.cc b/gcc/gimple-pretty-print.cc
index ebd87b20a0adc080c4a8f9429e75f49b96e72f9a..2a1a5b7f811ca341e8ee7e85a9701d3a37ff80bf 100644
--- a/gcc/gimple-pretty-print.cc
+++ b/gcc/gimple-pretty-print.cc
@@ -459,10 +459,6 @@ dump_binary_rhs (pretty_printer *buffer, const gassign *gs, int spc,
case VEC_PACK_FLOAT_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_SERIES_EXPR:
for (p = get_tree_code_name (code); *p; p++)
pp_character (buffer, TOUPPER (*p));
diff --git a/gcc/optabs-tree.cc b/gcc/optabs-tree.cc
index 8383fe820b080f6d66f83dcf3b77d3c9f869f4bc..2f5f93dc6624f86f6b5618cf6e7aa2b508053a64 100644
--- a/gcc/optabs-tree.cc
+++ b/gcc/optabs-tree.cc
@@ -190,22 +190,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
return (TYPE_UNSIGNED (type)
? vec_widen_ushiftl_lo_optab : vec_widen_sshiftl_lo_optab);
- case VEC_WIDEN_PLUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_lo_optab : vec_widen_saddl_lo_optab);
-
- case VEC_WIDEN_PLUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_hi_optab : vec_widen_saddl_hi_optab);
-
- case VEC_WIDEN_MINUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_lo_optab : vec_widen_ssubl_lo_optab);
-
- case VEC_WIDEN_MINUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_hi_optab : vec_widen_ssubl_hi_optab);
-
case VEC_UNPACK_HI_EXPR:
return (TYPE_UNSIGNED (type)
? vec_unpacku_hi_optab : vec_unpacks_hi_optab);
@@ -312,8 +296,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
'hi'/'lo' pair using codes such as VEC_WIDEN_MINUS_HI/LO.
Supported widening operations:
- WIDEN_MINUS_EXPR
- WIDEN_PLUS_EXPR
WIDEN_MULT_EXPR
WIDEN_LSHIFT_EXPR
@@ -345,12 +327,6 @@ supportable_half_widening_operation (enum tree_code code, tree vectype_out,
case WIDEN_LSHIFT_EXPR:
*code1 = LSHIFT_EXPR;
break;
- case WIDEN_MINUS_EXPR:
- *code1 = MINUS_EXPR;
- break;
- case WIDEN_PLUS_EXPR:
- *code1 = PLUS_EXPR;
- break;
case WIDEN_MULT_EXPR:
*code1 = MULT_EXPR;
break;
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index 8de1b144a426776bf464765477c71ee8f2e52b81..46eed1e1f22052fc077f2fc25e5be627bce541b6 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -3948,8 +3948,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case PLUS_EXPR:
case MINUS_EXPR:
{
@@ -4070,10 +4068,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 043e1d5987a4c4b0159109dafb85a805ca828c1e..c0bebb7f4de36838341ed62389ad0e2b79f03034 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4288,8 +4288,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case REALIGN_LOAD_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case DOT_PROD_EXPR:
@@ -4298,10 +4296,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case WIDEN_MULT_MINUS_EXPR:
case WIDEN_LSHIFT_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-pretty-print.cc b/gcc/tree-pretty-print.cc
index 6acd394a0790ad2ad989f195a3288f0f0a8cc489..53ca62dc1a6873ae9365f199061bde9edd486196 100644
--- a/gcc/tree-pretty-print.cc
+++ b/gcc/tree-pretty-print.cc
@@ -2825,8 +2825,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
break;
/* Binary arithmetic and logic expressions. */
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case MULT_EXPR:
@@ -3790,10 +3788,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
case VEC_SERIES_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
@@ -4311,12 +4305,6 @@ op_symbol_code (enum tree_code code)
case WIDEN_LSHIFT_EXPR:
return "w<<";
- case WIDEN_PLUS_EXPR:
- return "w+";
-
- case WIDEN_MINUS_EXPR:
- return "w-";
-
case POINTER_PLUS_EXPR:
return "+";
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index d20a10a1524164eef788ab4b88ba57c7a09c3387..98dd56ff022233ccead36a1f5a5e896e352f9f5b 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -136,8 +136,6 @@ vect_get_smallest_scalar_type (stmt_vec_info stmt_info, tree scalar_type)
|| gimple_assign_rhs_code (assign) == WIDEN_SUM_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_MULT_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_LSHIFT_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_PLUS_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_MINUS_EXPR
|| gimple_assign_rhs_code (assign) == FLOAT_EXPR)
{
tree rhs_type = TREE_TYPE (gimple_assign_rhs1 (assign));
@@ -3172,8 +3170,8 @@ vect_analyze_data_ref_accesses (vec_info *vinfo,
break;
/* Check that the DR_INITs are compile-time constants. */
- if (!tree_fits_shwi_p (DR_INIT (dra))
- || !tree_fits_shwi_p (DR_INIT (drb)))
+ if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST
+ || TREE_CODE (DR_INIT (drb)) != INTEGER_CST)
break;
/* Different .GOMP_SIMD_LANE calls still give the same lane,
@@ -3225,7 +3223,7 @@ vect_analyze_data_ref_accesses (vec_info *vinfo,
unsigned HOST_WIDE_INT step
= absu_hwi (tree_to_shwi (DR_STEP (dra)));
if (step != 0
- && step <= ((unsigned HOST_WIDE_INT)init_b - init_a))
+ && step <= (unsigned HOST_WIDE_INT)(init_b - init_a))
break;
}
}
diff --git a/gcc/tree-vect-generic.cc b/gcc/tree-vect-generic.cc
index 92aba5d4af61dd478ec3f1b94854e4ad84166774..5823b08baf70b89b22ecc148b0702a84671ad084 100644
--- a/gcc/tree-vect-generic.cc
+++ b/gcc/tree-vect-generic.cc
@@ -2209,10 +2209,6 @@ expand_vector_operations_1 (gimple_stmt_iterator *gsi,
arguments, not the widened result. VEC_UNPACK_FLOAT_*_EXPR is
calculated in the same way above. */
if (code == WIDEN_SUM_EXPR
- || code == VEC_WIDEN_PLUS_HI_EXPR
- || code == VEC_WIDEN_PLUS_LO_EXPR
- || code == VEC_WIDEN_MINUS_HI_EXPR
- || code == VEC_WIDEN_MINUS_LO_EXPR
|| code == VEC_WIDEN_MULT_HI_EXPR
|| code == VEC_WIDEN_MULT_LO_EXPR
|| code == VEC_WIDEN_MULT_EVEN_EXPR
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 268f5402fcdd5ec5bfb806db8c410e701c771275..64a0bde05bfcf62d2dc4bd18a9b6f1cb5f8698b5 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -559,21 +559,29 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else
+ rhs_code = gimple_call_combined_fn (stmt);
+
+ if (rhs_code.safe_as_tree_code () != code
+ && rhs_code.get_rep () != widened_code.get_rep ())
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = is_gimple_assign (stmt) ? gimple_assign_lhs (stmt):
+ gimple_call_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -586,7 +594,11 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op;
+ if (is_gimple_assign (stmt))
+ op = gimple_op (stmt, i + 1);
+ else
+ op = gimple_call_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1299,8 +1311,9 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
- false, 2, unprom, &half_type))
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ CFN_VEC_WIDEN_MINUS, false, 2, unprom,
+ &half_type))
return NULL;
vect_pattern_detected ("vect_recog_sad_pattern", last_stmt);
@@ -2339,9 +2352,10 @@ vect_recog_average_pattern (vec_info *vinfo,
internal_fn ifn = IFN_AVG_FLOOR;
vect_unpromoted_value unprom[3];
tree new_type;
+ enum optab_subtype subtype;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
- unprom, &new_type);
+ CFN_VEC_WIDEN_PLUS, false, 3,
+ unprom, &new_type, &subtype);
if (nops == 0)
return NULL;
if (nops == 3)
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 9af0d107fdafb959db10d87e4e0ba5fda4e47bd7..a2c0947864a64aea7c19a9c2bc53961821838e00 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4898,9 +4898,7 @@ vectorizable_conversion (vec_info *vinfo,
if (res_op.code.is_tree_code ())
{
- widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
+ widen_arith = (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR);
}
else
@@ -4954,8 +4952,6 @@ vectorizable_conversion (vec_info *vinfo,
{
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
|| code == IFN_VEC_WIDEN_PLUS
|| code == IFN_VEC_WIDEN_MINUS);
@@ -11981,7 +11977,7 @@ supportable_widening_operation (vec_info *vinfo,
class loop *vect_loop = NULL;
machine_mode vec_mode;
enum insn_code icode1, icode2;
- optab optab1, optab2;
+ optab optab1 = unknown_optab, optab2 = unknown_optab;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
code_helper c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
@@ -12075,16 +12071,6 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_WIDEN_LSHIFT_HI_EXPR;
break;
- case WIDEN_PLUS_EXPR:
- c1 = VEC_WIDEN_PLUS_LO_EXPR;
- c2 = VEC_WIDEN_PLUS_HI_EXPR;
- break;
-
- case WIDEN_MINUS_EXPR:
- c1 = VEC_WIDEN_MINUS_LO_EXPR;
- c2 = VEC_WIDEN_MINUS_HI_EXPR;
- break;
-
CASE_CONVERT:
c1 = VEC_UNPACK_LO_EXPR;
c2 = VEC_UNPACK_HI_EXPR;
diff --git a/gcc/tree.def b/gcc/tree.def
index 62650b6934b337c5d56e5393dc114173d72c9aa9..9b2dce3576440c445d3240b9ed937fe67c9a5992 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1383,8 +1383,6 @@ DEFTREECODE (WIDEN_MULT_MINUS_EXPR, "widen_mult_minus_expr", tcc_expression, 3)
the first argument from type t1 to type t2, and then shifting it
by the second argument. */
DEFTREECODE (WIDEN_LSHIFT_EXPR, "widen_lshift_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_PLUS_EXPR, "widen_plus_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_MINUS_EXPR, "widen_minus_expr", tcc_binary, 2)
/* Widening vector multiplication.
The two operands are vectors with N elements of size S. Multiplying the
@@ -1449,10 +1447,6 @@ DEFTREECODE (VEC_PACK_FLOAT_EXPR, "vec_pack_float_expr", tcc_binary, 2)
*/
DEFTREECODE (VEC_WIDEN_LSHIFT_HI_EXPR, "widen_lshift_hi_expr", tcc_binary, 2)
DEFTREECODE (VEC_WIDEN_LSHIFT_LO_EXPR, "widen_lshift_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_HI_EXPR, "widen_plus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_LO_EXPR, "widen_plus_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_HI_EXPR, "widen_minus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_LO_EXPR, "widen_minus_lo_expr", tcc_binary, 2)
/* PREDICT_EXPR. Specify hint for branch prediction. The
PREDICT_EXPR_PREDICTOR specify predictor and PREDICT_EXPR_OUTCOME the
--
2.17.1
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-06 17:20 ` Joel Hutton
@ 2022-06-07 8:18 ` Richard Sandiford
2022-06-07 9:01 ` Joel Hutton
2022-06-13 9:18 ` [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns Richard Biener
0 siblings, 2 replies; 53+ messages in thread
From: Richard Sandiford @ 2022-06-07 8:18 UTC (permalink / raw)
To: Joel Hutton; +Cc: Richard Biener, gcc-patches
Joel Hutton <Joel.Hutton@arm.com> writes:
>> > Patches attached. They already incorporated the .cc rename, now
>> > rebased to be after the change to tree.h
>>
>> @@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
>> 2, oprnd, half_type, unprom, vectype);
>>
>> tree var = vect_recog_temp_ssa_var (itype, NULL);
>> - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
>> - oprnd[0], oprnd[1]);
>> + gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0],
>> oprnd[1]);
>>
>>
>> you should be able to do without the new gimple_build overload
>> by using
>>
>> gimple_seq stmts = NULL;
>> gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
>> gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
>>
>> because 'gimple_build' is an existing API.
>
> Done.
>
> The gimple_build overload was at the request of Richard Sandiford, I assume removing it is ok with you Richard S?
> From Richard Sandiford:
>> For example, I think we should hide this inside a new:
>>
>> gimple_build (var, wide_code, oprnd[0], oprnd[1]);
>>
>> that works directly on code_helper, similarly to the new code_helper
>> gimple_build interfaces.
I thought the potential problem with the above is that gimple_build
is a folding interface, so in principle it's allowed to return an
existing SSA_NAME set by an existing statement (or even a constant).
I think in this context we do need to force a new statement to be
created.
Of course, the hope is that there wouldn't still be such folding
opportunities at this stage, but I don't think it's guaranteed
(especially with options fuzzing).
Sind I was mentioned :-) ...
Could you run the patch through contrib/check_GNU_style.py?
There seem to be a few long lines.
> + if (res_op.code.is_tree_code ())
Do you need this is_tree_code ()? These comparisons…
> + {
> + widen_arith = (code == WIDEN_PLUS_EXPR
> + || code == WIDEN_MINUS_EXPR
> + || code == WIDEN_MULT_EXPR
> + || code == WIDEN_LSHIFT_EXPR);
…ought to be safe unconditionally.
> + }
> + else
> + widen_arith = false;
> +
> + if (!widen_arith
> + && !CONVERT_EXPR_CODE_P (code)
> + && code != FIX_TRUNC_EXPR
> + && code != FLOAT_EXPR)
> + return false;
>
> /* Check types of lhs and rhs. */
> - scalar_dest = gimple_assign_lhs (stmt);
> + scalar_dest = gimple_get_lhs (stmt);
> lhs_type = TREE_TYPE (scalar_dest);
> vectype_out = STMT_VINFO_VECTYPE (stmt_info);
>
> @@ -4938,10 +4951,14 @@ vectorizable_conversion (vec_info *vinfo,
>
> if (op_type == binary_op)
> {
> - gcc_assert (code == WIDEN_MULT_EXPR || code == WIDEN_LSHIFT_EXPR
> - || code == WIDEN_PLUS_EXPR || code == WIDEN_MINUS_EXPR);
> + gcc_assert (code == WIDEN_MULT_EXPR
> + || code == WIDEN_LSHIFT_EXPR
> + || code == WIDEN_PLUS_EXPR
> + || code == WIDEN_MINUS_EXPR);
>
> - op1 = gimple_assign_rhs2 (stmt);
> +
> + op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
> + gimple_call_arg (stmt, 0);
> tree vectype1_in;
> if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
> &op1, &slp_op1, &dt[1], &vectype1_in))
> […]
> @@ -12181,7 +12235,6 @@ supportable_widening_operation (vec_info *vinfo,
> return false;
> }
>
> -
> /* Function supportable_narrowing_operation
>
> Check whether an operation represented by the code CODE is a
Seems like a spurious change.
> @@ -12205,7 +12258,7 @@ supportable_widening_operation (vec_info *vinfo,
> bool
> supportable_narrowing_operation (enum tree_code code,
> tree vectype_out, tree vectype_in,
> - enum tree_code *code1, int *multi_step_cvt,
> + tree_code* _code1, int *multi_step_cvt,
The original formatting (space before the “*”) was correct.
Names beginning with _ are reserved, so I think we need a different
name here. Also, the name in the comment should stay in sync with
the name in the code.
That said though, I'm not sure…
> vec<tree> *interm_types)
> {
> machine_mode vec_mode;
> @@ -12217,8 +12270,8 @@ supportable_narrowing_operation (enum tree_code code,
> tree intermediate_type, prev_type;
> machine_mode intermediate_mode, prev_mode;
> int i;
> - unsigned HOST_WIDE_INT n_elts;
> bool uns;
> + tree_code * code1 = (tree_code*) _code1;
…the combination of these two changes makes sense on their own.
>
> *multi_step_cvt = 0;
> switch (code)
> @@ -12227,9 +12280,8 @@ supportable_narrowing_operation (enum tree_code code,
> c1 = VEC_PACK_TRUNC_EXPR;
> if (VECTOR_BOOLEAN_TYPE_P (narrow_vectype)
> && VECTOR_BOOLEAN_TYPE_P (vectype)
> - && SCALAR_INT_MODE_P (TYPE_MODE (vectype))
> - && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&n_elts)
> - && n_elts < BITS_PER_UNIT)
> + && TYPE_MODE (narrow_vectype) == TYPE_MODE (vectype)
> + && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
> optab1 = vec_pack_sbool_trunc_optab;
> else
> optab1 = optab_for_tree_code (c1, vectype, optab_default);
> @@ -12320,9 +12372,8 @@ supportable_narrowing_operation (enum tree_code code,
> = lang_hooks.types.type_for_mode (intermediate_mode, uns);
> if (VECTOR_BOOLEAN_TYPE_P (intermediate_type)
> && VECTOR_BOOLEAN_TYPE_P (prev_type)
> - && SCALAR_INT_MODE_P (prev_mode)
> - && TYPE_VECTOR_SUBPARTS (intermediate_type).is_constant (&n_elts)
> - && n_elts < BITS_PER_UNIT)
> + && intermediate_mode == prev_mode
> + && SCALAR_INT_MODE_P (prev_mode))
> interm_optab = vec_pack_sbool_trunc_optab;
> else
> interm_optab
This part looks like a behavioural change, so I think it should be part
of a separate patch.
> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> index 642eb0aeb21264cd736a479b1ec25357abef29cd..50ff8eeac1e6b9859302bd784f10ee3d8ff4b4dc 100644
> --- a/gcc/tree-vectorizer.h
> +++ b/gcc/tree-vectorizer.h
> @@ -2120,13 +2120,12 @@ extern bool vect_is_simple_use (vec_info *, stmt_vec_info, slp_tree,
> enum vect_def_type *,
> tree *, stmt_vec_info * = NULL);
> extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
> -extern bool supportable_widening_operation (vec_info *,
> - enum tree_code, stmt_vec_info,
> - tree, tree, enum tree_code *,
> - enum tree_code *, int *,
> - vec<tree> *);
> +extern bool supportable_widening_operation (vec_info*, code_helper,
> + stmt_vec_info, tree, tree,
> + code_helper*, code_helper*,
> + int*, vec<tree> *);
> extern bool supportable_narrowing_operation (enum tree_code, tree, tree,
> - enum tree_code *, int *,
> + tree_code *, int *,
> vec<tree> *);
>
> extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
> diff --git a/gcc/tree.h b/gcc/tree.h
> index f84958933d51144bb6ce7cc41eca5f7f06814550..00b0e4d1c696633fe38baad5295b1f90398d53fc 100644
> --- a/gcc/tree.h
> +++ b/gcc/tree.h
> @@ -92,6 +92,10 @@ public:
> bool is_fn_code () const { return rep < 0; }
> bool is_internal_fn () const;
> bool is_builtin_fn () const;
> + enum tree_code safe_as_tree_code () const { return is_tree_code () ?
> + (tree_code)* this : MAX_TREE_CODES; }
> + combined_fn safe_as_fn_code () const { return is_fn_code () ? (combined_fn) *this
> + : CFN_LAST;}
Since these don't fit on a line, the coding convention says that they
should be defined outside of the class.
Thanks,
Richard
> int get_rep () const { return rep; }
> bool operator== (const code_helper &other) { return rep == other.rep; }
> bool operator!= (const code_helper &other) { return rep != other.rep; }
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-07 8:18 ` Richard Sandiford
@ 2022-06-07 9:01 ` Joel Hutton
2022-06-09 14:03 ` Joel Hutton
2022-06-13 9:02 ` Richard Biener
2022-06-13 9:18 ` [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns Richard Biener
1 sibling, 2 replies; 53+ messages in thread
From: Joel Hutton @ 2022-06-07 9:01 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Richard Biener, gcc-patches
Thanks Richard,
> I thought the potential problem with the above is that gimple_build is a
> folding interface, so in principle it's allowed to return an existing SSA_NAME
> set by an existing statement (or even a constant).
> I think in this context we do need to force a new statement to be created.
Before I make any changes, I'd like to check we're all on the same page.
richi, are you ok with the gimple_build function, perhaps with a different name if you are concerned with overloading? we could use gimple_ch_build or gimple_code_helper_build?
Similarly are you ok with the use of gimple_extract_op? I would lean towards using it as it is cleaner, but I don't have strong feelings.
Joel
> -----Original Message-----
> From: Richard Sandiford <richard.sandiford@arm.com>
> Sent: 07 June 2022 09:18
> To: Joel Hutton <Joel.Hutton@arm.com>
> Cc: Richard Biener <rguenther@suse.de>; gcc-patches@gcc.gnu.org
> Subject: Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as
> internal_fns
>
> Joel Hutton <Joel.Hutton@arm.com> writes:
> >> > Patches attached. They already incorporated the .cc rename, now
> >> > rebased to be after the change to tree.h
> >>
> >> @@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
> >> 2, oprnd, half_type, unprom, vectype);
> >>
> >> tree var = vect_recog_temp_ssa_var (itype, NULL);
> >> - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
> >> - oprnd[0], oprnd[1]);
> >> + gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0],
> >> oprnd[1]);
> >>
> >>
> >> you should be able to do without the new gimple_build overload by
> >> using
> >>
> >> gimple_seq stmts = NULL;
> >> gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
> >> gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
> >>
> >> because 'gimple_build' is an existing API.
> >
> > Done.
> >
> > The gimple_build overload was at the request of Richard Sandiford, I
> assume removing it is ok with you Richard S?
> > From Richard Sandiford:
> >> For example, I think we should hide this inside a new:
> >>
> >> gimple_build (var, wide_code, oprnd[0], oprnd[1]);
> >>
> >> that works directly on code_helper, similarly to the new code_helper
> >> gimple_build interfaces.
>
> I thought the potential problem with the above is that gimple_build is a
> folding interface, so in principle it's allowed to return an existing SSA_NAME
> set by an existing statement (or even a constant).
> I think in this context we do need to force a new statement to be created.
>
> Of course, the hope is that there wouldn't still be such folding opportunities
> at this stage, but I don't think it's guaranteed (especially with options
> fuzzing).
>
> Sind I was mentioned :-) ...
>
> Could you run the patch through contrib/check_GNU_style.py?
> There seem to be a few long lines.
>
> > + if (res_op.code.is_tree_code ())
>
> Do you need this is_tree_code ()? These comparisons…
>
> > + {
> > + widen_arith = (code == WIDEN_PLUS_EXPR
> > + || code == WIDEN_MINUS_EXPR
> > + || code == WIDEN_MULT_EXPR
> > + || code == WIDEN_LSHIFT_EXPR);
>
> …ought to be safe unconditionally.
>
> > + }
> > + else
> > + widen_arith = false;
> > +
> > + if (!widen_arith
> > + && !CONVERT_EXPR_CODE_P (code)
> > + && code != FIX_TRUNC_EXPR
> > + && code != FLOAT_EXPR)
> > + return false;
> >
> > /* Check types of lhs and rhs. */
> > - scalar_dest = gimple_assign_lhs (stmt);
> > + scalar_dest = gimple_get_lhs (stmt);
> > lhs_type = TREE_TYPE (scalar_dest);
> > vectype_out = STMT_VINFO_VECTYPE (stmt_info);
> >
> > @@ -4938,10 +4951,14 @@ vectorizable_conversion (vec_info *vinfo,
> >
> > if (op_type == binary_op)
> > {
> > - gcc_assert (code == WIDEN_MULT_EXPR || code ==
> WIDEN_LSHIFT_EXPR
> > - || code == WIDEN_PLUS_EXPR || code ==
> WIDEN_MINUS_EXPR);
> > + gcc_assert (code == WIDEN_MULT_EXPR
> > + || code == WIDEN_LSHIFT_EXPR
> > + || code == WIDEN_PLUS_EXPR
> > + || code == WIDEN_MINUS_EXPR);
> >
> > - op1 = gimple_assign_rhs2 (stmt);
> > +
> > + op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
> > + gimple_call_arg (stmt, 0);
> > tree vectype1_in;
> > if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
> > &op1, &slp_op1, &dt[1], &vectype1_in)) […] @@
> -12181,7
> > +12235,6 @@ supportable_widening_operation (vec_info *vinfo,
> > return false;
> > }
> >
> > -
> > /* Function supportable_narrowing_operation
> >
> > Check whether an operation represented by the code CODE is a
>
> Seems like a spurious change.
>
> > @@ -12205,7 +12258,7 @@ supportable_widening_operation (vec_info
> > *vinfo, bool supportable_narrowing_operation (enum tree_code code,
> > tree vectype_out, tree vectype_in,
> > - enum tree_code *code1, int *multi_step_cvt,
> > + tree_code* _code1, int *multi_step_cvt,
>
> The original formatting (space before the “*”) was correct.
> Names beginning with _ are reserved, so I think we need a different
> name here. Also, the name in the comment should stay in sync with
> the name in the code.
>
> That said though, I'm not sure…
>
> > vec<tree> *interm_types)
> > {
> > machine_mode vec_mode;
> > @@ -12217,8 +12270,8 @@ supportable_narrowing_operation (enum
> tree_code code,
> > tree intermediate_type, prev_type;
> > machine_mode intermediate_mode, prev_mode;
> > int i;
> > - unsigned HOST_WIDE_INT n_elts;
> > bool uns;
> > + tree_code * code1 = (tree_code*) _code1;
>
> …the combination of these two changes makes sense on their own.
>
> >
> > *multi_step_cvt = 0;
> > switch (code)
> > @@ -12227,9 +12280,8 @@ supportable_narrowing_operation (enum
> tree_code code,
> > c1 = VEC_PACK_TRUNC_EXPR;
> > if (VECTOR_BOOLEAN_TYPE_P (narrow_vectype)
> > && VECTOR_BOOLEAN_TYPE_P (vectype)
> > - && SCALAR_INT_MODE_P (TYPE_MODE (vectype))
> > - && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&n_elts)
> > - && n_elts < BITS_PER_UNIT)
> > + && TYPE_MODE (narrow_vectype) == TYPE_MODE (vectype)
> > + && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
> > optab1 = vec_pack_sbool_trunc_optab;
> > else
> > optab1 = optab_for_tree_code (c1, vectype, optab_default);
> > @@ -12320,9 +12372,8 @@ supportable_narrowing_operation (enum
> tree_code code,
> > = lang_hooks.types.type_for_mode (intermediate_mode, uns);
> > if (VECTOR_BOOLEAN_TYPE_P (intermediate_type)
> > && VECTOR_BOOLEAN_TYPE_P (prev_type)
> > - && SCALAR_INT_MODE_P (prev_mode)
> > - && TYPE_VECTOR_SUBPARTS (intermediate_type).is_constant
> (&n_elts)
> > - && n_elts < BITS_PER_UNIT)
> > + && intermediate_mode == prev_mode
> > + && SCALAR_INT_MODE_P (prev_mode))
> > interm_optab = vec_pack_sbool_trunc_optab;
> > else
> > interm_optab
>
> This part looks like a behavioural change, so I think it should be part
> of a separate patch.
>
> > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> > index
> 642eb0aeb21264cd736a479b1ec25357abef29cd..50ff8eeac1e6b9859302bd78
> 4f10ee3d8ff4b4dc 100644
> > --- a/gcc/tree-vectorizer.h
> > +++ b/gcc/tree-vectorizer.h
> > @@ -2120,13 +2120,12 @@ extern bool vect_is_simple_use (vec_info *,
> stmt_vec_info, slp_tree,
> > enum vect_def_type *,
> > tree *, stmt_vec_info * = NULL);
> > extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
> > -extern bool supportable_widening_operation (vec_info *,
> > - enum tree_code, stmt_vec_info,
> > - tree, tree, enum tree_code *,
> > - enum tree_code *, int *,
> > - vec<tree> *);
> > +extern bool supportable_widening_operation (vec_info*, code_helper,
> > + stmt_vec_info, tree, tree,
> > + code_helper*, code_helper*,
> > + int*, vec<tree> *);
> > extern bool supportable_narrowing_operation (enum tree_code, tree,
> tree,
> > - enum tree_code *, int *,
> > + tree_code *, int *,
> > vec<tree> *);
> >
> > extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
> > diff --git a/gcc/tree.h b/gcc/tree.h
> > index
> f84958933d51144bb6ce7cc41eca5f7f06814550..00b0e4d1c696633fe38baad5
> 295b1f90398d53fc 100644
> > --- a/gcc/tree.h
> > +++ b/gcc/tree.h
> > @@ -92,6 +92,10 @@ public:
> > bool is_fn_code () const { return rep < 0; }
> > bool is_internal_fn () const;
> > bool is_builtin_fn () const;
> > + enum tree_code safe_as_tree_code () const { return is_tree_code () ?
> > + (tree_code)* this : MAX_TREE_CODES; }
> > + combined_fn safe_as_fn_code () const { return is_fn_code () ?
> (combined_fn) *this
> > + : CFN_LAST;}
>
> Since these don't fit on a line, the coding convention says that they
> should be defined outside of the class.
>
> Thanks,
> Richard
>
> > int get_rep () const { return rep; }
> > bool operator== (const code_helper &other) { return rep == other.rep; }
> > bool operator!= (const code_helper &other) { return rep != other.rep; }
^ permalink raw reply [flat|nested] 53+ messages in thread
* [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-07 9:01 ` Joel Hutton
@ 2022-06-09 14:03 ` Joel Hutton
2022-06-13 9:02 ` Richard Biener
1 sibling, 0 replies; 53+ messages in thread
From: Joel Hutton @ 2022-06-09 14:03 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Richard Biener, gcc-patches
> Before I make any changes, I'd like to check we're all on the same page.
>
> richi, are you ok with the gimple_build function, perhaps with a different
> name if you are concerned with overloading? we could use gimple_ch_build
> or gimple_code_helper_build?
>
> Similarly are you ok with the use of gimple_extract_op? I would lean towards
> using it as it is cleaner, but I don't have strong feelings.
>
> Joel
Ping. Just looking for some confirmation before I rework this patch. It would be good to get some agreement on this as Tamar is blocked on this patch.
Joel
> -----Original Message-----
> From: Joel Hutton
> Sent: 07 June 2022 10:02
> To: Richard Sandiford <richard.sandiford@arm.com>
> Cc: Richard Biener <rguenther@suse.de>; gcc-patches@gcc.gnu.org
> Subject: RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as
> internal_fns
>
> Thanks Richard,
>
> > I thought the potential problem with the above is that gimple_build is
> > a folding interface, so in principle it's allowed to return an
> > existing SSA_NAME set by an existing statement (or even a constant).
> > I think in this context we do need to force a new statement to be created.
>
> Before I make any changes, I'd like to check we're all on the same page.
>
> richi, are you ok with the gimple_build function, perhaps with a different
> name if you are concerned with overloading? we could use gimple_ch_build
> or gimple_code_helper_build?
>
> Similarly are you ok with the use of gimple_extract_op? I would lean towards
> using it as it is cleaner, but I don't have strong feelings.
>
> Joel
>
> > -----Original Message-----
> > From: Richard Sandiford <richard.sandiford@arm.com>
> > Sent: 07 June 2022 09:18
> > To: Joel Hutton <Joel.Hutton@arm.com>
> > Cc: Richard Biener <rguenther@suse.de>; gcc-patches@gcc.gnu.org
> > Subject: Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as
> > internal_fns
> >
> > Joel Hutton <Joel.Hutton@arm.com> writes:
> > >> > Patches attached. They already incorporated the .cc rename, now
> > >> > rebased to be after the change to tree.h
> > >>
> > >> @@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info
> *vinfo,
> > >> 2, oprnd, half_type, unprom, vectype);
> > >>
> > >> tree var = vect_recog_temp_ssa_var (itype, NULL);
> > >> - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
> > >> - oprnd[0], oprnd[1]);
> > >> + gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0],
> > >> oprnd[1]);
> > >>
> > >>
> > >> you should be able to do without the new gimple_build overload by
> > >> using
> > >>
> > >> gimple_seq stmts = NULL;
> > >> gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
> > >> gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
> > >>
> > >> because 'gimple_build' is an existing API.
> > >
> > > Done.
> > >
> > > The gimple_build overload was at the request of Richard Sandiford, I
> > assume removing it is ok with you Richard S?
> > > From Richard Sandiford:
> > >> For example, I think we should hide this inside a new:
> > >>
> > >> gimple_build (var, wide_code, oprnd[0], oprnd[1]);
> > >>
> > >> that works directly on code_helper, similarly to the new
> > >> code_helper gimple_build interfaces.
> >
> > I thought the potential problem with the above is that gimple_build is
> > a folding interface, so in principle it's allowed to return an
> > existing SSA_NAME set by an existing statement (or even a constant).
> > I think in this context we do need to force a new statement to be created.
> >
> > Of course, the hope is that there wouldn't still be such folding
> > opportunities at this stage, but I don't think it's guaranteed
> > (especially with options fuzzing).
> >
> > Sind I was mentioned :-) ...
> >
> > Could you run the patch through contrib/check_GNU_style.py?
> > There seem to be a few long lines.
> >
> > > + if (res_op.code.is_tree_code ())
> >
> > Do you need this is_tree_code ()? These comparisons…
> >
> > > + {
> > > + widen_arith = (code == WIDEN_PLUS_EXPR
> > > + || code == WIDEN_MINUS_EXPR
> > > + || code == WIDEN_MULT_EXPR
> > > + || code == WIDEN_LSHIFT_EXPR);
> >
> > …ought to be safe unconditionally.
> >
> > > + }
> > > + else
> > > + widen_arith = false;
> > > +
> > > + if (!widen_arith
> > > + && !CONVERT_EXPR_CODE_P (code)
> > > + && code != FIX_TRUNC_EXPR
> > > + && code != FLOAT_EXPR)
> > > + return false;
> > >
> > > /* Check types of lhs and rhs. */
> > > - scalar_dest = gimple_assign_lhs (stmt);
> > > + scalar_dest = gimple_get_lhs (stmt);
> > > lhs_type = TREE_TYPE (scalar_dest);
> > > vectype_out = STMT_VINFO_VECTYPE (stmt_info);
> > >
> > > @@ -4938,10 +4951,14 @@ vectorizable_conversion (vec_info *vinfo,
> > >
> > > if (op_type == binary_op)
> > > {
> > > - gcc_assert (code == WIDEN_MULT_EXPR || code ==
> > WIDEN_LSHIFT_EXPR
> > > - || code == WIDEN_PLUS_EXPR || code ==
> > WIDEN_MINUS_EXPR);
> > > + gcc_assert (code == WIDEN_MULT_EXPR
> > > + || code == WIDEN_LSHIFT_EXPR
> > > + || code == WIDEN_PLUS_EXPR
> > > + || code == WIDEN_MINUS_EXPR);
> > >
> > > - op1 = gimple_assign_rhs2 (stmt);
> > > +
> > > + op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
> > > + gimple_call_arg (stmt, 0);
> > > tree vectype1_in;
> > > if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
> > > &op1, &slp_op1, &dt[1], &vectype1_in)) […] @@
> > -12181,7
> > > +12235,6 @@ supportable_widening_operation (vec_info *vinfo,
> > > return false;
> > > }
> > >
> > > -
> > > /* Function supportable_narrowing_operation
> > >
> > > Check whether an operation represented by the code CODE is a
> >
> > Seems like a spurious change.
> >
> > > @@ -12205,7 +12258,7 @@ supportable_widening_operation (vec_info
> > > *vinfo, bool supportable_narrowing_operation (enum tree_code code,
> > > tree vectype_out, tree vectype_in,
> > > - enum tree_code *code1, int *multi_step_cvt,
> > > + tree_code* _code1, int *multi_step_cvt,
> >
> > The original formatting (space before the “*”) was correct.
> > Names beginning with _ are reserved, so I think we need a different
> > name here. Also, the name in the comment should stay in sync with the
> > name in the code.
> >
> > That said though, I'm not sure…
> >
> > > vec<tree> *interm_types) {
> > > machine_mode vec_mode;
> > > @@ -12217,8 +12270,8 @@ supportable_narrowing_operation (enum
> > tree_code code,
> > > tree intermediate_type, prev_type;
> > > machine_mode intermediate_mode, prev_mode;
> > > int i;
> > > - unsigned HOST_WIDE_INT n_elts;
> > > bool uns;
> > > + tree_code * code1 = (tree_code*) _code1;
> >
> > …the combination of these two changes makes sense on their own.
> >
> > >
> > > *multi_step_cvt = 0;
> > > switch (code)
> > > @@ -12227,9 +12280,8 @@ supportable_narrowing_operation (enum
> > tree_code code,
> > > c1 = VEC_PACK_TRUNC_EXPR;
> > > if (VECTOR_BOOLEAN_TYPE_P (narrow_vectype)
> > > && VECTOR_BOOLEAN_TYPE_P (vectype)
> > > - && SCALAR_INT_MODE_P (TYPE_MODE (vectype))
> > > - && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&n_elts)
> > > - && n_elts < BITS_PER_UNIT)
> > > + && TYPE_MODE (narrow_vectype) == TYPE_MODE (vectype)
> > > + && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
> > > optab1 = vec_pack_sbool_trunc_optab;
> > > else
> > > optab1 = optab_for_tree_code (c1, vectype, optab_default); @@
> > > -12320,9 +12372,8 @@ supportable_narrowing_operation (enum
> > tree_code code,
> > > = lang_hooks.types.type_for_mode (intermediate_mode, uns);
> > > if (VECTOR_BOOLEAN_TYPE_P (intermediate_type)
> > > && VECTOR_BOOLEAN_TYPE_P (prev_type)
> > > - && SCALAR_INT_MODE_P (prev_mode)
> > > - && TYPE_VECTOR_SUBPARTS (intermediate_type).is_constant
> > (&n_elts)
> > > - && n_elts < BITS_PER_UNIT)
> > > + && intermediate_mode == prev_mode
> > > + && SCALAR_INT_MODE_P (prev_mode))
> > > interm_optab = vec_pack_sbool_trunc_optab;
> > > else
> > > interm_optab
> >
> > This part looks like a behavioural change, so I think it should be
> > part of a separate patch.
> >
> > > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index
> >
> 642eb0aeb21264cd736a479b1ec25357abef29cd..50ff8eeac1e6b9859302bd78
> > 4f10ee3d8ff4b4dc 100644
> > > --- a/gcc/tree-vectorizer.h
> > > +++ b/gcc/tree-vectorizer.h
> > > @@ -2120,13 +2120,12 @@ extern bool vect_is_simple_use (vec_info *,
> > stmt_vec_info, slp_tree,
> > > enum vect_def_type *,
> > > tree *, stmt_vec_info * = NULL); extern bool
> > > vect_maybe_update_slp_op_vectype (slp_tree, tree); -extern bool
> > > supportable_widening_operation (vec_info *,
> > > - enum tree_code, stmt_vec_info,
> > > - tree, tree, enum tree_code *,
> > > - enum tree_code *, int *,
> > > - vec<tree> *);
> > > +extern bool supportable_widening_operation (vec_info*, code_helper,
> > > + stmt_vec_info, tree, tree,
> > > + code_helper*, code_helper*,
> > > + int*, vec<tree> *);
> > > extern bool supportable_narrowing_operation (enum tree_code, tree,
> > tree,
> > > - enum tree_code *, int *,
> > > + tree_code *, int *,
> > > vec<tree> *);
> > >
> > > extern unsigned record_stmt_cost (stmt_vector_for_cost *, int, diff
> > > --git a/gcc/tree.h b/gcc/tree.h index
> >
> f84958933d51144bb6ce7cc41eca5f7f06814550..00b0e4d1c696633fe38baad5
> > 295b1f90398d53fc 100644
> > > --- a/gcc/tree.h
> > > +++ b/gcc/tree.h
> > > @@ -92,6 +92,10 @@ public:
> > > bool is_fn_code () const { return rep < 0; }
> > > bool is_internal_fn () const;
> > > bool is_builtin_fn () const;
> > > + enum tree_code safe_as_tree_code () const { return is_tree_code () ?
> > > + (tree_code)* this : MAX_TREE_CODES; } combined_fn
> > > + safe_as_fn_code () const { return is_fn_code () ?
> > (combined_fn) *this
> > > + : CFN_LAST;}
> >
> > Since these don't fit on a line, the coding convention says that they
> > should be defined outside of the class.
> >
> > Thanks,
> > Richard
> >
> > > int get_rep () const { return rep; }
> > > bool operator== (const code_helper &other) { return rep == other.rep; }
> > > bool operator!= (const code_helper &other) { return rep !=
> > > other.rep; }
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-07 9:01 ` Joel Hutton
2022-06-09 14:03 ` Joel Hutton
@ 2022-06-13 9:02 ` Richard Biener
2022-06-30 13:20 ` Joel Hutton
1 sibling, 1 reply; 53+ messages in thread
From: Richard Biener @ 2022-06-13 9:02 UTC (permalink / raw)
To: Joel Hutton; +Cc: Richard Sandiford, gcc-patches
On Tue, 7 Jun 2022, Joel Hutton wrote:
> Thanks Richard,
>
> > I thought the potential problem with the above is that gimple_build is a
> > folding interface, so in principle it's allowed to return an existing SSA_NAME
> > set by an existing statement (or even a constant).
> > I think in this context we do need to force a new statement to be created.
>
> Before I make any changes, I'd like to check we're all on the same page.
>
> richi, are you ok with the gimple_build function, perhaps with a
> different name if you are concerned with overloading? we could use
> gimple_ch_build or gimple_code_helper_build?
We can go with a private vect_gimple_build function until we sort out
the API issue to unblock Tamar (I'll reply to Richards reply with further
thoughts on this)
> Similarly are you ok with the use of gimple_extract_op? I would lean towards using it as it is cleaner, but I don't have strong feelings.
I don't like using gimple_extract_op here, I think I outlined a variant
that is even shorter.
Richard.
> Joel
>
> > -----Original Message-----
> > From: Richard Sandiford <richard.sandiford@arm.com>
> > Sent: 07 June 2022 09:18
> > To: Joel Hutton <Joel.Hutton@arm.com>
> > Cc: Richard Biener <rguenther@suse.de>; gcc-patches@gcc.gnu.org
> > Subject: Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as
> > internal_fns
> >
> > Joel Hutton <Joel.Hutton@arm.com> writes:
> > >> > Patches attached. They already incorporated the .cc rename, now
> > >> > rebased to be after the change to tree.h
> > >>
> > >> @@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
> > >> 2, oprnd, half_type, unprom, vectype);
> > >>
> > >> tree var = vect_recog_temp_ssa_var (itype, NULL);
> > >> - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
> > >> - oprnd[0], oprnd[1]);
> > >> + gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0],
> > >> oprnd[1]);
> > >>
> > >>
> > >> you should be able to do without the new gimple_build overload by
> > >> using
> > >>
> > >> gimple_seq stmts = NULL;
> > >> gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
> > >> gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
> > >>
> > >> because 'gimple_build' is an existing API.
> > >
> > > Done.
> > >
> > > The gimple_build overload was at the request of Richard Sandiford, I
> > assume removing it is ok with you Richard S?
> > > From Richard Sandiford:
> > >> For example, I think we should hide this inside a new:
> > >>
> > >> gimple_build (var, wide_code, oprnd[0], oprnd[1]);
> > >>
> > >> that works directly on code_helper, similarly to the new code_helper
> > >> gimple_build interfaces.
> >
> > I thought the potential problem with the above is that gimple_build is a
> > folding interface, so in principle it's allowed to return an existing SSA_NAME
> > set by an existing statement (or even a constant).
> > I think in this context we do need to force a new statement to be created.
> >
> > Of course, the hope is that there wouldn't still be such folding opportunities
> > at this stage, but I don't think it's guaranteed (especially with options
> > fuzzing).
> >
> > Sind I was mentioned :-) ...
> >
> > Could you run the patch through contrib/check_GNU_style.py?
> > There seem to be a few long lines.
> >
> > > + if (res_op.code.is_tree_code ())
> >
> > Do you need this is_tree_code ()? These comparisons…
> >
> > > + {
> > > + widen_arith = (code == WIDEN_PLUS_EXPR
> > > + || code == WIDEN_MINUS_EXPR
> > > + || code == WIDEN_MULT_EXPR
> > > + || code == WIDEN_LSHIFT_EXPR);
> >
> > …ought to be safe unconditionally.
> >
> > > + }
> > > + else
> > > + widen_arith = false;
> > > +
> > > + if (!widen_arith
> > > + && !CONVERT_EXPR_CODE_P (code)
> > > + && code != FIX_TRUNC_EXPR
> > > + && code != FLOAT_EXPR)
> > > + return false;
> > >
> > > /* Check types of lhs and rhs. */
> > > - scalar_dest = gimple_assign_lhs (stmt);
> > > + scalar_dest = gimple_get_lhs (stmt);
> > > lhs_type = TREE_TYPE (scalar_dest);
> > > vectype_out = STMT_VINFO_VECTYPE (stmt_info);
> > >
> > > @@ -4938,10 +4951,14 @@ vectorizable_conversion (vec_info *vinfo,
> > >
> > > if (op_type == binary_op)
> > > {
> > > - gcc_assert (code == WIDEN_MULT_EXPR || code ==
> > WIDEN_LSHIFT_EXPR
> > > - || code == WIDEN_PLUS_EXPR || code ==
> > WIDEN_MINUS_EXPR);
> > > + gcc_assert (code == WIDEN_MULT_EXPR
> > > + || code == WIDEN_LSHIFT_EXPR
> > > + || code == WIDEN_PLUS_EXPR
> > > + || code == WIDEN_MINUS_EXPR);
> > >
> > > - op1 = gimple_assign_rhs2 (stmt);
> > > +
> > > + op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
> > > + gimple_call_arg (stmt, 0);
> > > tree vectype1_in;
> > > if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
> > > &op1, &slp_op1, &dt[1], &vectype1_in)) […] @@
> > -12181,7
> > > +12235,6 @@ supportable_widening_operation (vec_info *vinfo,
> > > return false;
> > > }
> > >
> > > -
> > > /* Function supportable_narrowing_operation
> > >
> > > Check whether an operation represented by the code CODE is a
> >
> > Seems like a spurious change.
> >
> > > @@ -12205,7 +12258,7 @@ supportable_widening_operation (vec_info
> > > *vinfo, bool supportable_narrowing_operation (enum tree_code code,
> > > tree vectype_out, tree vectype_in,
> > > - enum tree_code *code1, int *multi_step_cvt,
> > > + tree_code* _code1, int *multi_step_cvt,
> >
> > The original formatting (space before the “*”) was correct.
> > Names beginning with _ are reserved, so I think we need a different
> > name here. Also, the name in the comment should stay in sync with
> > the name in the code.
> >
> > That said though, I'm not sure…
> >
> > > vec<tree> *interm_types)
> > > {
> > > machine_mode vec_mode;
> > > @@ -12217,8 +12270,8 @@ supportable_narrowing_operation (enum
> > tree_code code,
> > > tree intermediate_type, prev_type;
> > > machine_mode intermediate_mode, prev_mode;
> > > int i;
> > > - unsigned HOST_WIDE_INT n_elts;
> > > bool uns;
> > > + tree_code * code1 = (tree_code*) _code1;
> >
> > …the combination of these two changes makes sense on their own.
> >
> > >
> > > *multi_step_cvt = 0;
> > > switch (code)
> > > @@ -12227,9 +12280,8 @@ supportable_narrowing_operation (enum
> > tree_code code,
> > > c1 = VEC_PACK_TRUNC_EXPR;
> > > if (VECTOR_BOOLEAN_TYPE_P (narrow_vectype)
> > > && VECTOR_BOOLEAN_TYPE_P (vectype)
> > > - && SCALAR_INT_MODE_P (TYPE_MODE (vectype))
> > > - && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&n_elts)
> > > - && n_elts < BITS_PER_UNIT)
> > > + && TYPE_MODE (narrow_vectype) == TYPE_MODE (vectype)
> > > + && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
> > > optab1 = vec_pack_sbool_trunc_optab;
> > > else
> > > optab1 = optab_for_tree_code (c1, vectype, optab_default);
> > > @@ -12320,9 +12372,8 @@ supportable_narrowing_operation (enum
> > tree_code code,
> > > = lang_hooks.types.type_for_mode (intermediate_mode, uns);
> > > if (VECTOR_BOOLEAN_TYPE_P (intermediate_type)
> > > && VECTOR_BOOLEAN_TYPE_P (prev_type)
> > > - && SCALAR_INT_MODE_P (prev_mode)
> > > - && TYPE_VECTOR_SUBPARTS (intermediate_type).is_constant
> > (&n_elts)
> > > - && n_elts < BITS_PER_UNIT)
> > > + && intermediate_mode == prev_mode
> > > + && SCALAR_INT_MODE_P (prev_mode))
> > > interm_optab = vec_pack_sbool_trunc_optab;
> > > else
> > > interm_optab
> >
> > This part looks like a behavioural change, so I think it should be part
> > of a separate patch.
> >
> > > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
> > > index
> > 642eb0aeb21264cd736a479b1ec25357abef29cd..50ff8eeac1e6b9859302bd78
> > 4f10ee3d8ff4b4dc 100644
> > > --- a/gcc/tree-vectorizer.h
> > > +++ b/gcc/tree-vectorizer.h
> > > @@ -2120,13 +2120,12 @@ extern bool vect_is_simple_use (vec_info *,
> > stmt_vec_info, slp_tree,
> > > enum vect_def_type *,
> > > tree *, stmt_vec_info * = NULL);
> > > extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
> > > -extern bool supportable_widening_operation (vec_info *,
> > > - enum tree_code, stmt_vec_info,
> > > - tree, tree, enum tree_code *,
> > > - enum tree_code *, int *,
> > > - vec<tree> *);
> > > +extern bool supportable_widening_operation (vec_info*, code_helper,
> > > + stmt_vec_info, tree, tree,
> > > + code_helper*, code_helper*,
> > > + int*, vec<tree> *);
> > > extern bool supportable_narrowing_operation (enum tree_code, tree,
> > tree,
> > > - enum tree_code *, int *,
> > > + tree_code *, int *,
> > > vec<tree> *);
> > >
> > > extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
> > > diff --git a/gcc/tree.h b/gcc/tree.h
> > > index
> > f84958933d51144bb6ce7cc41eca5f7f06814550..00b0e4d1c696633fe38baad5
> > 295b1f90398d53fc 100644
> > > --- a/gcc/tree.h
> > > +++ b/gcc/tree.h
> > > @@ -92,6 +92,10 @@ public:
> > > bool is_fn_code () const { return rep < 0; }
> > > bool is_internal_fn () const;
> > > bool is_builtin_fn () const;
> > > + enum tree_code safe_as_tree_code () const { return is_tree_code () ?
> > > + (tree_code)* this : MAX_TREE_CODES; }
> > > + combined_fn safe_as_fn_code () const { return is_fn_code () ?
> > (combined_fn) *this
> > > + : CFN_LAST;}
> >
> > Since these don't fit on a line, the coding convention says that they
> > should be defined outside of the class.
> >
> > Thanks,
> > Richard
> >
> > > int get_rep () const { return rep; }
> > > bool operator== (const code_helper &other) { return rep == other.rep; }
> > > bool operator!= (const code_helper &other) { return rep != other.rep; }
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-07 8:18 ` Richard Sandiford
2022-06-07 9:01 ` Joel Hutton
@ 2022-06-13 9:18 ` Richard Biener
1 sibling, 0 replies; 53+ messages in thread
From: Richard Biener @ 2022-06-13 9:18 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Joel Hutton, gcc-patches
On Tue, 7 Jun 2022, Richard Sandiford wrote:
> Joel Hutton <Joel.Hutton@arm.com> writes:
> >> > Patches attached. They already incorporated the .cc rename, now
> >> > rebased to be after the change to tree.h
> >>
> >> @@ -1412,8 +1412,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
> >> 2, oprnd, half_type, unprom, vectype);
> >>
> >> tree var = vect_recog_temp_ssa_var (itype, NULL);
> >> - gimple *pattern_stmt = gimple_build_assign (var, wide_code,
> >> - oprnd[0], oprnd[1]);
> >> + gimple *pattern_stmt = gimple_build (var, wide_code, oprnd[0],
> >> oprnd[1]);
> >>
> >>
> >> you should be able to do without the new gimple_build overload
> >> by using
> >>
> >> gimple_seq stmts = NULL;
> >> gimple_build (&stmts, wide_code, itype, oprnd[0], oprnd[1]);
> >> gimple *pattern_stmt = gimple_seq_last_stmt (stmts);
> >>
> >> because 'gimple_build' is an existing API.
> >
> > Done.
> >
> > The gimple_build overload was at the request of Richard Sandiford, I assume removing it is ok with you Richard S?
> > From Richard Sandiford:
> >> For example, I think we should hide this inside a new:
> >>
> >> gimple_build (var, wide_code, oprnd[0], oprnd[1]);
> >>
> >> that works directly on code_helper, similarly to the new code_helper
> >> gimple_build interfaces.
>
> I thought the potential problem with the above is that gimple_build
> is a folding interface, so in principle it's allowed to return an
> existing SSA_NAME set by an existing statement (or even a constant).
> I think in this context we do need to force a new statement to be
> created.
Yes, that's due to how we use vect_finish_stmt_generation (only?).
It might be useful to add an overload that takes a gimple_seq
instead of a single gimple * for the vectorized stmt and leave
all the magic to that. Now - we have the additional issue
that we have STMT_VINFO_VEC_STMTS instead of STMT_VINFO_VEC_DEFS
(in the end we'll only ever need the defs, never the stmts I think).
I do think that we eventually want to 'enhance' the gimple.h
non-folding stmt building API, unfortunately I took the 'gimple_build'
name for the folding one, so alternatively we can unify assign/call
with gimple_build_assign_or_call (...). I don't really like the
idea of having folding and non-folding APIs being overloads :/
Maybe the non-folding API should be CTORs (guess GTY won't like
that) or static member functions:
gimple *gimple::build (tree, code_helper, tree, tree);
and in the long run the gimple_build API should be (for some uses?)
off a class as well, like instead of
gimple_seq seq = NULL;
op = gimple_build (&seq, ...);
do
gimple_builder b (location); // location defaulted to UNKNOWN
op = b.build (...);
So - writing the above I somewhat like the idea of static
member functions in 'gimple' (yes, at the root of the class
hierarchy, definitely not at gimple_statement_with_memory_ops_base,
not sure if we want gassign::build for assigns
and only the code_helper 'overloads' at the class root).
Richard.
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-13 9:02 ` Richard Biener
@ 2022-06-30 13:20 ` Joel Hutton
2022-07-12 12:32 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Joel Hutton @ 2022-06-30 13:20 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Sandiford, gcc-patches, Andre Simoes Dias Vieira
[-- Attachment #1: Type: text/plain, Size: 647 bytes --]
> We can go with a private vect_gimple_build function until we sort out the API
> issue to unblock Tamar (I'll reply to Richards reply with further thoughts on
> this)
>
Done.
> > Similarly are you ok with the use of gimple_extract_op? I would lean
> towards using it as it is cleaner, but I don't have strong feelings.
>
> I don't like using gimple_extract_op here, I think I outlined a variant that is
> even shorter.
>
Done.
Updated patches attached, bootstrapped and regression tested on aarch64.
Tomorrow is my last working day at Arm, so it will likely be Andre that commits this/addresses any further comments.
[-- Attachment #2: 0001-Refactor-to-allow-internal_fn-s.patch --]
[-- Type: application/octet-stream, Size: 23390 bytes --]
From f1321c617838e94044cbae357a63db002fbd3edb Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Wed, 25 Aug 2021 14:31:15 +0100
Subject: [PATCH 1/3] Refactor to allow internal_fn's
Hi all,
This refactor allows widening patterns (such as widen_plus/widen_minus) to be represented as
either internal_fns or tree_codes.
[vect-patterns] Refactor as internal_fn's
Refactor vect-patterns to allow patterns to be internal_fns starting
with widening_plus/minus patterns
gcc/ChangeLog:
* tree-core.h (ECF_WIDEN): New flag.
* gimple-match.h (class code_helper): * tree-core.h (ECF_WIDEN): Flag to mark internal_fn as widening.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Refactor to
use code_helper.
(vect_gimple_build): New function.
* tree-vect-stmts.cc (vect_gen_widened_results_half): Refactor to
use code_helper.
(vect_create_vectorized_promotion_stmts): Refactor to use
code_helper.
(vectorizable_conversion): Refactor to use code_helper.
gimple_call or gimple_assign.
(supportable_widening_operation): Refactor to use code_helper.
(supportable_narrowing_operation): Refactor to use code_helper.
* tree-vectorizer.h (supportable_widening_operation): Change
prototype to use code_helper.
(supportable_narrowing_operation): change prototype to use
code_helper.
(vect_gimple_build): New function prototype.
* tree.h (code_helper::safe_as_tree_code): New function.
helper functions.
(code_helper::safe_as_fn_code): New function.
---
gcc/tree-core.h | 3 +
gcc/tree-vect-patterns.cc | 34 ++++++-
gcc/tree-vect-stmts.cc | 208 +++++++++++++++++++++++++-------------
gcc/tree-vectorizer.h | 14 +--
gcc/tree.h | 13 +++
5 files changed, 189 insertions(+), 83 deletions(-)
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index ab5fa01e5cb5fb56c1964b93b014ed55a4aa704a..cff6211080bced0bffb39e98039a6550897acf77 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -96,6 +96,9 @@ struct die_struct;
/* Nonzero if this is a cold function. */
#define ECF_COLD (1 << 15)
+/* Nonzero if this is a widening function. */
+#define ECF_WIDEN (1 << 16)
+
/* Call argument flags. */
/* Nonzero if the argument is not used by the function. */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 8f624863971392c891fde7278949c8818f646576..d892158f024fc045b897aebe76f2e2b66211cf83 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -25,6 +25,8 @@ along with GCC; see the file COPYING3. If not see
#include "rtl.h"
#include "tree.h"
#include "gimple.h"
+#include "gimple-iterator.h"
+#include "gimple-fold.h"
#include "ssa.h"
#include "expmed.h"
#include "optabs-tree.h"
@@ -1348,7 +1350,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
- tree_code orig_code, tree_code wide_code,
+ tree_code orig_code, code_helper wide_code,
bool shift_p, const char *name)
{
gimple *last_stmt = last_stmt_info->stmt;
@@ -1391,7 +1393,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
vecctype = get_vectype_for_scalar_type (vinfo, ctype);
}
- enum tree_code dummy_code;
+ code_helper dummy_code;
int dummy_int;
auto_vec<tree> dummy_vec;
if (!vectype
@@ -1412,8 +1414,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
2, oprnd, half_type, unprom, vectype);
tree var = vect_recog_temp_ssa_var (itype, NULL);
- gimple *pattern_stmt = gimple_build_assign (var, wide_code,
- oprnd[0], oprnd[1]);
+ gimple *pattern_stmt = vect_gimple_build (var, wide_code, oprnd[0], oprnd[1]);
if (vecctype != vecitype)
pattern_stmt = vect_convert_output (vinfo, last_stmt_info, ctype,
@@ -5968,3 +5969,28 @@ vect_pattern_recog (vec_info *vinfo)
/* After this no more add_stmt calls are allowed. */
vinfo->stmt_vec_info_ro = true;
}
+
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple *
+vect_gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
+{
+ if (op0 == NULL_TREE)
+ return NULL;
+ if (ch.is_tree_code ())
+ return op1 == NULL_TREE ? gimple_build_assign (lhs, ch.safe_as_tree_code (),
+ op0) :
+ gimple_build_assign (lhs, ch.safe_as_tree_code (),
+ op0, op1);
+ else
+ {
+ internal_fn fn = as_internal_fn (ch.safe_as_fn_code ());
+ gimple* stmt;
+ if (op1 == NULL_TREE)
+ stmt = gimple_build_call_internal (fn, 1, op0);
+ else
+ stmt = gimple_build_call_internal (fn, 2, op0, op1);
+ gimple_call_set_lhs (stmt, lhs);
+ return stmt;
+ }
+}
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 346d8ce280437e00bfeb19a4b4adc59eb96207f9..d6aabb873c86ab8ff0bae41c7f6c3bad34d583c5 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4636,7 +4636,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
STMT_INFO is the original scalar stmt that we are vectorizing. */
static gimple *
-vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
+vect_gen_widened_results_half (vec_info *vinfo, code_helper ch,
tree vec_oprnd0, tree vec_oprnd1, int op_type,
tree vec_dest, gimple_stmt_iterator *gsi,
stmt_vec_info stmt_info)
@@ -4645,12 +4645,11 @@ vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
tree new_temp;
/* Generate half of the widened result: */
- gcc_assert (op_type == TREE_CODE_LENGTH (code));
if (op_type != binary_op)
vec_oprnd1 = NULL;
- new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0, vec_oprnd1);
+ new_stmt = vect_gimple_build (vec_dest, ch, vec_oprnd0, vec_oprnd1);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
return new_stmt;
@@ -4729,8 +4728,8 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
- enum tree_code code2, int op_type)
+ code_helper ch1,
+ code_helper ch2, int op_type)
{
int i;
tree vop0, vop1, new_tmp1, new_tmp2;
@@ -4746,10 +4745,10 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vop1 = NULL_TREE;
/* Generate the two halves of promotion operation. */
- new_stmt1 = vect_gen_widened_results_half (vinfo, code1, vop0, vop1,
+ new_stmt1 = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
- new_stmt2 = vect_gen_widened_results_half (vinfo, code2, vop0, vop1,
+ new_stmt2 = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
if (is_gimple_call (new_stmt1))
@@ -4846,8 +4845,9 @@ vectorizable_conversion (vec_info *vinfo,
tree scalar_dest;
tree op0, op1 = NULL_TREE;
loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
- enum tree_code code, code1 = ERROR_MARK, code2 = ERROR_MARK;
- enum tree_code codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
+ tree_code tc1;
+ code_helper code, code1, code2;
+ code_helper codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
tree new_temp;
enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type};
int ndts = 2;
@@ -4876,31 +4876,43 @@ vectorizable_conversion (vec_info *vinfo,
&& ! vec_stmt)
return false;
- gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
- if (!stmt)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return false;
- if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
+ if (gimple_get_lhs (stmt) == NULL_TREE
+ || TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- code = gimple_assign_rhs_code (stmt);
- if (!CONVERT_EXPR_CODE_P (code)
- && code != FIX_TRUNC_EXPR
- && code != FLOAT_EXPR
- && code != WIDEN_PLUS_EXPR
- && code != WIDEN_MINUS_EXPR
- && code != WIDEN_MULT_EXPR
- && code != WIDEN_LSHIFT_EXPR)
+ if (TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
+ return false;
+
+ if (is_gimple_assign (stmt))
+ {
+ code = gimple_assign_rhs_code (stmt);
+ op_type = TREE_CODE_LENGTH (code.safe_as_tree_code ());
+ }
+ else if (gimple_call_internal_p (stmt))
+ {
+ code = gimple_call_internal_fn (stmt);
+ op_type = gimple_call_num_args (stmt);
+ }
+ else
return false;
bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
- op_type = TREE_CODE_LENGTH (code);
+ || code == WIDEN_MINUS_EXPR
+ || code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR);
+
+ if (!widen_arith
+ && !CONVERT_EXPR_CODE_P (code)
+ && code != FIX_TRUNC_EXPR
+ && code != FLOAT_EXPR)
+ return false;
/* Check types of lhs and rhs. */
- scalar_dest = gimple_assign_lhs (stmt);
+ scalar_dest = gimple_get_lhs (stmt);
lhs_type = TREE_TYPE (scalar_dest);
vectype_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -4938,10 +4950,14 @@ vectorizable_conversion (vec_info *vinfo,
if (op_type == binary_op)
{
- gcc_assert (code == WIDEN_MULT_EXPR || code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR || code == WIDEN_MINUS_EXPR);
+ gcc_assert (code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR
+ || code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR);
- op1 = gimple_assign_rhs2 (stmt);
+
+ op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
+ gimple_call_arg (stmt, 0);
tree vectype1_in;
if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
&op1, &slp_op1, &dt[1], &vectype1_in))
@@ -5025,8 +5041,12 @@ vectorizable_conversion (vec_info *vinfo,
&& code != FLOAT_EXPR
&& !CONVERT_EXPR_CODE_P (code))
return false;
- if (supportable_convert_operation (code, vectype_out, vectype_in, &code1))
+ if (supportable_convert_operation (code.safe_as_tree_code (), vectype_out,
+ vectype_in, &tc1))
+ {
+ code1 = tc1;
break;
+ }
/* FALLTHRU */
unsupported:
if (dump_enabled_p ())
@@ -5037,9 +5057,11 @@ vectorizable_conversion (vec_info *vinfo,
case WIDEN:
if (known_eq (nunits_in, nunits_out))
{
- if (!supportable_half_widening_operation (code, vectype_out,
- vectype_in, &code1))
+ if (!supportable_half_widening_operation (code.safe_as_tree_code (),
+ vectype_out, vectype_in,
+ &tc1))
goto unsupported;
+ code1 = tc1;
gcc_assert (!(multi_step_cvt && op_type == binary_op));
break;
}
@@ -5073,14 +5095,17 @@ vectorizable_conversion (vec_info *vinfo,
if (GET_MODE_SIZE (rhs_mode) == fltsz)
{
- if (!supportable_convert_operation (code, vectype_out,
- cvt_type, &codecvt1))
+ tc1 = ERROR_MARK;
+ if (!supportable_convert_operation (code.safe_as_tree_code (),
+ vectype_out,
+ cvt_type, &tc1))
goto unsupported;
+ codecvt1 = tc1;
}
- else if (!supportable_widening_operation (vinfo, code, stmt_info,
- vectype_out, cvt_type,
- &codecvt1, &codecvt2,
- &multi_step_cvt,
+ else if (!supportable_widening_operation (vinfo, code,
+ stmt_info, vectype_out,
+ cvt_type, &codecvt1,
+ &codecvt2, &multi_step_cvt,
&interm_types))
continue;
else
@@ -5088,8 +5113,9 @@ vectorizable_conversion (vec_info *vinfo,
if (supportable_widening_operation (vinfo, NOP_EXPR, stmt_info,
cvt_type,
- vectype_in, &code1, &code2,
- &multi_step_cvt, &interm_types))
+ vectype_in, &code1,
+ &code2, &multi_step_cvt,
+ &interm_types))
{
found_mode = true;
break;
@@ -5111,10 +5137,15 @@ vectorizable_conversion (vec_info *vinfo,
case NARROW:
gcc_assert (op_type == unary_op);
- if (supportable_narrowing_operation (code, vectype_out, vectype_in,
- &code1, &multi_step_cvt,
+ if (supportable_narrowing_operation (code.safe_as_tree_code (),
+ vectype_out,
+ vectype_in,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
if (code != FIX_TRUNC_EXPR
|| GET_MODE_SIZE (lhs_mode) >= GET_MODE_SIZE (rhs_mode))
@@ -5125,13 +5156,18 @@ vectorizable_conversion (vec_info *vinfo,
cvt_type = get_same_sized_vectype (cvt_type, vectype_in);
if (cvt_type == NULL_TREE)
goto unsupported;
- if (!supportable_convert_operation (code, cvt_type, vectype_in,
- &codecvt1))
+ if (!supportable_convert_operation (code.safe_as_tree_code (), cvt_type,
+ vectype_in,
+ &tc1))
goto unsupported;
+ codecvt1 = tc1;
if (supportable_narrowing_operation (NOP_EXPR, vectype_out, cvt_type,
- &code1, &multi_step_cvt,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
goto unsupported;
default:
@@ -5245,8 +5281,10 @@ vectorizable_conversion (vec_info *vinfo,
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
/* Arguments are ready, create the new vector stmt. */
- gcc_assert (TREE_CODE_LENGTH (code1) == unary_op);
- gassign *new_stmt = gimple_build_assign (vec_dest, code1, vop0);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) code1) == unary_op);
+ gassign *new_stmt = gimple_build_assign (vec_dest,
+ code1.safe_as_tree_code (),
+ vop0);
new_temp = make_ssa_name (vec_dest, new_stmt);
gimple_assign_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
@@ -5278,7 +5316,7 @@ vectorizable_conversion (vec_info *vinfo,
for (i = multi_step_cvt; i >= 0; i--)
{
tree this_dest = vec_dsts[i];
- enum tree_code c1 = code1, c2 = code2;
+ code_helper c1 = code1, c2 = code2;
if (i == 0 && codecvt2 != ERROR_MARK)
{
c1 = codecvt1;
@@ -5288,7 +5326,8 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_half_widening_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
this_dest, gsi,
- c1, op_type);
+ c1.safe_as_tree_code (),
+ op_type);
else
vect_create_vectorized_promotion_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
@@ -5301,9 +5340,11 @@ vectorizable_conversion (vec_info *vinfo,
gimple *new_stmt;
if (cvt_type)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, codecvt1, vop0);
+ new_stmt = gimple_build_assign (new_temp,
+ codecvt1.safe_as_tree_code (),
+ vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -5327,10 +5368,12 @@ vectorizable_conversion (vec_info *vinfo,
if (cvt_type)
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH (((tree_code) codecvt1)) == unary_op);
new_temp = make_ssa_name (vec_dest);
gassign *new_stmt
- = gimple_build_assign (new_temp, codecvt1, vop0);
+ = gimple_build_assign (new_temp,
+ codecvt1.safe_as_tree_code (),
+ vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
vec_oprnds0[i] = new_temp;
}
@@ -5338,7 +5381,8 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_vectorized_demotion_stmts (vinfo, &vec_oprnds0,
multi_step_cvt,
stmt_info, vec_dsts, gsi,
- slp_node, code1);
+ slp_node,
+ code1.safe_as_tree_code ());
break;
}
if (!slp_node)
@@ -11926,9 +11970,11 @@ vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype)
bool
supportable_widening_operation (vec_info *vinfo,
- enum tree_code code, stmt_vec_info stmt_info,
+ code_helper code,
+ stmt_vec_info stmt_info,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, enum tree_code *code2,
+ code_helper *code1,
+ code_helper *code2,
int *multi_step_cvt,
vec<tree> *interm_types)
{
@@ -11939,7 +11985,7 @@ supportable_widening_operation (vec_info *vinfo,
optab optab1, optab2;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
- enum tree_code c1, c2;
+ code_helper c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
int i;
tree prev_type, intermediate_type;
machine_mode intermediate_mode, prev_mode;
@@ -11949,7 +11995,7 @@ supportable_widening_operation (vec_info *vinfo,
if (loop_info)
vect_loop = LOOP_VINFO_LOOP (loop_info);
- switch (code)
+ switch (code.safe_as_tree_code ())
{
case WIDEN_MULT_EXPR:
/* The result of a vectorized widening operation usually requires
@@ -11990,8 +12036,9 @@ supportable_widening_operation (vec_info *vinfo,
&& !nested_in_vect_loop_p (vect_loop, stmt_info)
&& supportable_widening_operation (vinfo, VEC_WIDEN_MULT_EVEN_EXPR,
stmt_info, vectype_out,
- vectype_in, code1, code2,
- multi_step_cvt, interm_types))
+ vectype_in, code1,
+ code2, multi_step_cvt,
+ interm_types))
{
/* Elements in a vector with vect_used_by_reduction property cannot
be reordered if the use chain with this property does not have the
@@ -12054,6 +12101,9 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_UNPACK_FIX_TRUNC_HI_EXPR;
break;
+ case MAX_TREE_CODES:
+ break;
+
default:
gcc_unreachable ();
}
@@ -12064,10 +12114,12 @@ supportable_widening_operation (vec_info *vinfo,
if (code == FIX_TRUNC_EXPR)
{
/* The signedness is determined from output operand. */
- optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
- optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype_out,
+ optab_default);
}
- else if (CONVERT_EXPR_CODE_P (code)
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
&& VECTOR_BOOLEAN_TYPE_P (wide_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
&& TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
@@ -12080,8 +12132,10 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype,
+ optab_default);
}
if (!optab1 || !optab2)
@@ -12092,8 +12146,12 @@ supportable_widening_operation (vec_info *vinfo,
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- *code1 = c1;
- *code2 = c2;
+ if (code.is_tree_code ())
+ {
+ *code1 = c1;
+ *code2 = c2;
+ }
+
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
@@ -12114,7 +12172,7 @@ supportable_widening_operation (vec_info *vinfo,
prev_type = vectype;
prev_mode = vec_mode;
- if (!CONVERT_EXPR_CODE_P (code))
+ if (!CONVERT_EXPR_CODE_P ((tree_code) code))
return false;
/* We assume here that there will not be more than MAX_INTERM_CVT_STEPS
@@ -12145,8 +12203,12 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab3 = optab_for_tree_code (c1, intermediate_type, optab_default);
- optab4 = optab_for_tree_code (c2, intermediate_type, optab_default);
+ optab3 = optab_for_tree_code (c1.safe_as_tree_code (),
+ intermediate_type,
+ optab_default);
+ optab4 = optab_for_tree_code (c2.safe_as_tree_code (),
+ intermediate_type,
+ optab_default);
}
if (!optab3 || !optab4
@@ -12205,7 +12267,7 @@ supportable_widening_operation (vec_info *vinfo,
bool
supportable_narrowing_operation (enum tree_code code,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, int *multi_step_cvt,
+ tree_code *code1, int *multi_step_cvt,
vec<tree> *interm_types)
{
machine_mode vec_mode;
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 642eb0aeb21264cd736a479b1ec25357abef29cd..6f70bd622c4a4dea8c432cd26c96d24af399ef3e 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2120,13 +2120,12 @@ extern bool vect_is_simple_use (vec_info *, stmt_vec_info, slp_tree,
enum vect_def_type *,
tree *, stmt_vec_info * = NULL);
extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
-extern bool supportable_widening_operation (vec_info *,
- enum tree_code, stmt_vec_info,
- tree, tree, enum tree_code *,
- enum tree_code *, int *,
- vec<tree> *);
+extern bool supportable_widening_operation (vec_info*, code_helper,
+ stmt_vec_info, tree, tree,
+ code_helper*, code_helper*,
+ int*, vec<tree> *);
extern bool supportable_narrowing_operation (enum tree_code, tree, tree,
- enum tree_code *, int *,
+ tree_code *, int *,
vec<tree> *);
extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
@@ -2558,4 +2557,7 @@ vect_is_integer_truncation (stmt_vec_info stmt_info)
&& TYPE_PRECISION (lhs_type) < TYPE_PRECISION (rhs_type));
}
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple * vect_gimple_build (tree, code_helper, tree, tree);
#endif /* GCC_TREE_VECTORIZER_H */
diff --git a/gcc/tree.h b/gcc/tree.h
index 6f6ad5a3a5f4dd4173482dfe259acf539ba24000..24b5184122550fe21ab0a5387867b6c65c20bb03 100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -93,6 +93,8 @@ public:
bool is_internal_fn () const;
bool is_builtin_fn () const;
int get_rep () const { return rep; }
+ enum tree_code safe_as_tree_code () const;
+ combined_fn safe_as_fn_code () const;
bool operator== (const code_helper &other) { return rep == other.rep; }
bool operator!= (const code_helper &other) { return rep != other.rep; }
bool operator== (tree_code c) { return rep == code_helper (c).rep; }
@@ -102,6 +104,17 @@ private:
int rep;
};
+inline enum tree_code
+code_helper::safe_as_tree_code () const
+{
+ return is_tree_code () ? (tree_code)* this : MAX_TREE_CODES;
+}
+
+inline combined_fn
+code_helper::safe_as_fn_code () const {
+ return is_fn_code () ? (combined_fn) *this : CFN_LAST;
+}
+
inline code_helper::operator internal_fn () const
{
return as_internal_fn (combined_fn (*this));
--
2.17.1
[-- Attachment #3: 0002-Refactor-widen_plus-as-internal_fn.patch --]
[-- Type: application/octet-stream, Size: 22832 bytes --]
From 1e8afa697157c3cb520a36304326e14891444226 Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Wed, 26 Jan 2022 14:00:17 +0000
Subject: [PATCH 2/3] Refactor widen_plus as internal_fn
This patch replaces the existing tree_code widen_plus and widen_minus
patterns with internal_fn versions.
DEF_INTERNAL_OPTAB_MULTI_FN is like DEF_INTERNAL_OPTAB_FN except it provides convenience wrappers for defining conversions that require a hi/lo split, like widening and narrowing operations. Each definition for <NAME> will require an optab named <OPTAB> and two other optabs that you specify for signed and unsigned. The hi/lo pair is necessary because the widening operations take n narrow elements as inputs and return n/2 wide elements as outputs. The 'lo' operation operates on the first n/2 elements of input. The 'hi' operation operates on the second n/2 elements of input. Defining an internal_fn along with hi/lo variations allows a single internal function to be returned from a vect_recog function that will later be expanded to hi/lo.
DEF_INTERNAL_OPTAB_MULTI_FN is used in internal-fn.def to register a widening internal_fn. It is defined differently in different places and internal-fn.def is sourced from those places so the parameters given can be reused.
internal-fn.c: defined to expand to hi/lo signed/unsigned optabs, later defined to generate the 'expand_' functions for the hi/lo versions of the fn.
internal-fn.def: defined to invoke DEF_INTERNAL_OPTAB_FN for the original and hi/lo variants of the internal_fn
For example:
IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>addl_hi_<mode> -> (u/s)addl2
IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>addl_lo_<mode> -> (u/s)addl
This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
gcc/ChangeLog:
2022-04-13 Joel Hutton <joel.hutton@arm.com>
2022-04-13 Tamar Christina <tamar.christina@arm.com>
* internal-fn.cc (INCLUDE_MAP): Include maps for use in optab
lookup.
(DEF_INTERNAL_OPTAB_MULTI_FN): Macro to define an internal_fn that
expands into multiple internal_fns (for widening).
(ifn_cmp): Function to compare ifn's for sorting/searching.
(lookup_multi_ifn_optab): Add lookup function.
(lookup_multi_internal_fn): Add lookup function.
(commutative_binary_fn_p): Add widen_plus fn's.
* internal-fn.def (DEF_INTERNAL_OPTAB_MULTI_FN): Define widening
plus,minus functions.
(VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS tree code.
(VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS tree code.
* internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
(lookup_multi_ifn_optab): Add prototype.
(lookup_multi_internal_fn): Add prototype.
* optabs.cc (commutative_optab_p): Add widening plus, minus optabs.
* optabs.def (OPTAB_CD): widen add, sub optabs
* tree-core.h (ECF_MULTI): Flag to indicate if a function decays
into hi/lo parts.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
patterns with a hi/lo split.
(vect_recog_widen_plus_pattern): Refactor to return
IFN_VECT_WIDEN_PLUS.
(vect_recog_widen_minus_pattern): Refactor to return new
IFN_VEC_WIDEN_MINUS.
* tree-vect-stmts.cc (vectorizable_conversion): Add widen plus/minus
ifn
support.
(supportable_widening_operation): Add widen plus/minus ifn support.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-widen-add.c: Test that new
IFN_VEC_WIDEN_PLUS is being used.
* gcc.target/aarch64/vect-widen-sub.c: Test that new
IFN_VEC_WIDEN_MINUS is being used.
---
gcc/internal-fn.cc | 107 ++++++++++++++++++
gcc/internal-fn.def | 23 ++++
gcc/internal-fn.h | 7 ++
gcc/optabs.cc | 12 +-
gcc/optabs.def | 2 +
.../gcc.target/aarch64/vect-widen-add.c | 4 +-
.../gcc.target/aarch64/vect-widen-sub.c | 4 +-
gcc/tree-core.h | 4 +
gcc/tree-vect-patterns.cc | 37 ++++--
gcc/tree-vect-stmts.cc | 68 +++++++++--
10 files changed, 249 insertions(+), 19 deletions(-)
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 91588f8bc9f7c3fe2bac17f3c4e6078cddb7b4d2..b2cb3e508027a84e4456d676d78b27b6c04b7b61 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -17,6 +17,7 @@ You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3. If not see
<http://www.gnu.org/licenses/>. */
+#define INCLUDE_MAP
#include "config.h"
#include "system.h"
#include "coretypes.h"
@@ -70,6 +71,26 @@ const int internal_fn_flags_array[] = {
0
};
+const enum internal_fn internal_fn_hilo_keys_array[] = {
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ IFN_##NAME##_LO, \
+ IFN_##NAME##_HI,
+#include "internal-fn.def"
+ IFN_LAST
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+};
+
+const optab internal_fn_hilo_values_array[] = {
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ SOPTAB##_lo_optab, UOPTAB##_lo_optab, \
+ SOPTAB##_hi_optab, UOPTAB##_hi_optab,
+#include "internal-fn.def"
+ unknown_optab, unknown_optab
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+};
+
/* Return the internal function called NAME, or IFN_LAST if there's
no such function. */
@@ -90,6 +111,62 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+static int
+ifn_cmp (const void *a_, const void *b_)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ auto *a = (const std::pair<ifn_pair, optab> *)a_;
+ auto *b = (const std::pair<ifn_pair, optab> *)b_;
+ return (int) (a->first.first) - (b->first.first);
+}
+
+/* Return the optab belonging to the given internal function NAME for the given
+ SIGN or unknown_optab. */
+
+optab
+lookup_multi_ifn_optab (enum internal_fn fn, unsigned sign)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ typedef auto_vec <std::pair<ifn_pair, optab>>fn_to_optab_map_type;
+ static fn_to_optab_map_type *fn_to_optab_map;
+
+ if (!fn_to_optab_map)
+ {
+ unsigned num
+ = sizeof (internal_fn_hilo_keys_array) / sizeof (enum internal_fn);
+ fn_to_optab_map = new fn_to_optab_map_type ();
+ for (unsigned int i = 0; i < num - 1; ++i)
+ {
+ enum internal_fn fn = internal_fn_hilo_keys_array[i];
+ optab v1 = internal_fn_hilo_values_array[2*i];
+ optab v2 = internal_fn_hilo_values_array[2*i + 1];
+ ifn_pair key1 (fn, 0);
+ fn_to_optab_map->safe_push ({key1, v1});
+ ifn_pair key2 (fn, 1);
+ fn_to_optab_map->safe_push ({key2, v2});
+ }
+ fn_to_optab_map->qsort (ifn_cmp);
+ }
+
+ ifn_pair new_pair (fn, sign ? 1 : 0);
+ optab tmp;
+ std::pair<ifn_pair,optab> pair_wrap (new_pair, tmp);
+ auto entry = fn_to_optab_map->bsearch (&pair_wrap, ifn_cmp);
+ return entry != fn_to_optab_map->end () ? entry->second : unknown_optab;
+}
+
+extern void
+lookup_multi_internal_fn (enum internal_fn ifn, enum internal_fn *lo,
+ enum internal_fn *hi)
+{
+ int ecf_flags = internal_fn_flags (ifn);
+ gcc_assert (ecf_flags & ECF_MULTI);
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -3928,6 +4005,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -4013,6 +4093,32 @@ set_edom_supported_p (void)
#endif
}
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(CODE, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ static void \
+ expand_##CODE (internal_fn, gcall *) \
+ { \
+ gcc_unreachable (); \
+ } \
+ static void \
+ expand_##CODE##_LO (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_lo##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_lo##_optab); \
+ } \
+ static void \
+ expand_##CODE##_HI (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_hi##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_hi##_optab); \
+ }
+
#define DEF_INTERNAL_OPTAB_FN(CODE, FLAGS, OPTAB, TYPE) \
static void \
expand_##CODE (internal_fn fn, gcall *stmt) \
@@ -4029,6 +4135,7 @@ set_edom_supported_p (void)
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_MULTI_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index d2d550d358606022b1cb44fa842f06e0be507bc3..4635a9c8af9ad27bb05d7510388d0fe2270428e5 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -82,6 +82,13 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_OPTAB_MULTI_FN is like DEF_INTERNAL_OPTAB_FN except it
+ provides convenience wrappers for defining conversions that require a
+ hi/lo split, like widening and narrowing operations. Each definition
+ for <NAME> will require an optab named <OPTAB> and two other optabs that
+ you specify for signed and unsigned.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -120,6 +127,14 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_OPTAB_MULTI_FN
+#define DEF_INTERNAL_OPTAB_MULTI_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME, FLAGS | ECF_MULTI, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, unknown, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, unknown, TYPE)
+#endif
+
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -292,6 +307,14 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_OPTAB_MULTI_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_WIDEN | ECF_NOTHROW,
+ vec_widen_add, vec_widen_saddl, vec_widen_uaddl,
+ binary)
+DEF_INTERNAL_OPTAB_MULTI_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_WIDEN | ECF_NOTHROW,
+ vec_widen_sub, vec_widen_ssubl, vec_widen_usubl,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 23c014a963c4d72da92c763db87ee486a2adb485..b35de19747d251d19dc13de1e0323368bd2ebdf2 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,9 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern optab lookup_multi_ifn_optab (enum internal_fn, unsigned);
+extern void lookup_multi_internal_fn (enum internal_fn, enum internal_fn *,
+ enum internal_fn *);
/* Return the ECF_* flags for function FN. */
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index a50dd798f2a454ac54e247f3e6cbab17577ea304..f9be369a6c5b99de5bbad664a11364d1c2cc4b95 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1312,7 +1312,17 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_add_optab
+ || binoptab == vec_widen_sub_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_ssubl_hi_optab
+ || binoptab == vec_widen_ssubl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab
+ || binoptab == vec_widen_usubl_hi_optab
+ || binoptab == vec_widen_usubl_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 801310ebaa7d469520809bb7efed6820f8eb866b..a7881dcb49e4ef07d8f07aa31214eb3a7a944e2e 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -78,6 +78,8 @@ OPTAB_CD(smsub_widen_optab, "msub$b$a4")
OPTAB_CD(umsub_widen_optab, "umsub$b$a4")
OPTAB_CD(ssmsub_widen_optab, "ssmsub$b$a4")
OPTAB_CD(usmsub_widen_optab, "usmsub$a$b4")
+OPTAB_CD(vec_widen_add_optab, "add$a$b3")
+OPTAB_CD(vec_widen_sub_optab, "sub$a$b3")
OPTAB_CD(vec_load_lanes_optab, "vec_load_lanes$a$b")
OPTAB_CD(vec_store_lanes_optab, "vec_store_lanes$a$b")
OPTAB_CD(vec_mask_load_lanes_optab, "vec_mask_load_lanes$a$b")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-core.h b/gcc/tree-core.h
index cff6211080bced0bffb39e98039a6550897acf77..d0c8b812cfb9c3ac83bf25fff0431b08cb7d823d 100644
--- a/gcc/tree-core.h
+++ b/gcc/tree-core.h
@@ -99,6 +99,10 @@ struct die_struct;
/* Nonzero if this is a widening function. */
#define ECF_WIDEN (1 << 16)
+/* Nonzero if this is a function that decomposes into a lo/hi operation. */
+#define ECF_MULTI (1 << 17)
+
+
/* Call argument flags. */
/* Nonzero if the argument is not used by the function. */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index d892158f024fc045b897aebe76f2e2b66211cf83..62ca28d725ed4ac8d7e4d493119e40772a0fbac6 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1351,14 +1351,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1424,6 +1426,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1437,26 +1453,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_popcount_pattern
@@ -5629,6 +5649,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index d6aabb873c86ab8ff0bae41c7f6c3bad34d583c5..6fa0669fdfc8630842b3f9f32f4b4a253e79bb92 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4903,7 +4903,9 @@ vectorizable_conversion (vec_info *vinfo,
bool widen_arith = (code == WIDEN_PLUS_EXPR
|| code == WIDEN_MINUS_EXPR
|| code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
+ || code == WIDEN_LSHIFT_EXPR
+ || code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -4953,7 +4955,9 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
+ || code == WIDEN_MINUS_EXPR
+ || code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
@@ -12130,14 +12134,62 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
- {
- optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype,
- optab_default);
- optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype,
- optab_default);
+
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn (code.safe_as_fn_code ());
+ int ecf_flags = internal_fn_flags (ifn);
+ gcc_assert (ecf_flags & ECF_MULTI);
+
+ switch (code.safe_as_fn_code ())
+ {
+ case CFN_VEC_WIDEN_PLUS:
+ break;
+ case CFN_VEC_WIDEN_MINUS:
+ break;
+ case CFN_LAST:
+ default:
+ return false;
+ }
+
+ internal_fn lo, hi;
+ lookup_multi_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
}
+ if (code.is_tree_code ())
+ {
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype_out,
+ optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype,
+ optab_default);
+ }
+ }
+
if (!optab1 || !optab2)
return false;
--
2.17.1
[-- Attachment #4: 0003-Remove-widen_plus-minus_expr-tree-codes.patch --]
[-- Type: application/octet-stream, Size: 19552 bytes --]
From 60664218e6e59510f02fb64b49a236e9e5b26c9f Mon Sep 17 00:00:00 2001
From: Joel Hutton <joel.hutton@arm.com>
Date: Fri, 28 Jan 2022 12:04:44 +0000
Subject: [PATCH 3/3] Remove widen_plus/minus_expr tree codes
This patch removes the old widen plus/minus tree codes which have been
replaced by internal functions.
gcc/ChangeLog:
* doc/generic.texi: Remove old tree codes.
* expr.cc (expand_expr_real_2): Remove old tree code cases.
* gimple-pretty-print.cc (dump_binary_rhs): Remove old tree code
cases.
* optabs-tree.cc (optab_for_tree_code): Remove old tree code cases.
(supportable_half_widening_operation): Remove old tree code cases.
* tree-cfg.cc (verify_gimple_assign_binary): Remove old tree code
cases.
* tree-inline.cc (estimate_operator_cost): Remove old tree code
cases.
* tree-pretty-print.cc (dump_generic_node): Remove tree code definition.
(op_symbol_code): Remove old tree code
cases.
* tree-vect-data-refs.cc (vect_get_smallest_scalar_type): Remove old tree code
cases.
(vect_analyze_data_ref_accesses): Remove old tree code
cases.
* tree-vect-generic.cc (expand_vector_operations_1): Remove old tree code
cases.
* tree-vect-patterns.cc (vect_widened_op_tree): Refactor ot replace
usage in vect_recog_sad_pattern.
(vect_recog_sad_pattern): Replace tree code widening pattern with
internal function.
(vect_recog_average_pattern): Replace tree code widening pattern
with internal function.
* tree-vect-stmts.cc (vectorizable_conversion): Remove old tree code
cases.
(supportable_widening_operation): Remove old tree code
cases.
* tree.def (WIDEN_PLUS_EXPR): Remove tree code definition.
(WIDEN_MINUS_EXPR): Remove tree code definition.
(VEC_WIDEN_PLUS_HI_EXPR): Remove tree code definition.
(VEC_WIDEN_PLUS_LO_EXPR): Remove tree code definition.
(VEC_WIDEN_MINUS_HI_EXPR): Remove tree code definition.
(VEC_WIDEN_MINUS_LO_EXPR): Remove tree code definition.
---
gcc/doc/generic.texi | 31 -------------------------------
gcc/expr.cc | 6 ------
gcc/gimple-pretty-print.cc | 4 ----
gcc/optabs-tree.cc | 24 ------------------------
gcc/tree-cfg.cc | 6 ------
gcc/tree-inline.cc | 6 ------
gcc/tree-pretty-print.cc | 12 ------------
gcc/tree-vect-data-refs.cc | 8 +++-----
gcc/tree-vect-generic.cc | 4 ----
gcc/tree-vect-patterns.cc | 36 +++++++++++++++++++++++++-----------
gcc/tree-vect-stmts.cc | 18 ++----------------
gcc/tree.def | 6 ------
12 files changed, 30 insertions(+), 131 deletions(-)
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index e5f9d1be8ea81f3da002ec3bb925590d331a2551..344045efd419b0cc3a11771acf70d2fd279c48ac 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,10 +1811,6 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1861,33 +1857,6 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
-@item VEC_WIDEN_PLUS_HI_EXPR
-@itemx VEC_WIDEN_PLUS_LO_EXPR
-These nodes represent widening vector addition of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The result
-is a vector that contains half as many elements, of an integral type whose size
-is twice as wide. In the case of @code{VEC_WIDEN_PLUS_HI_EXPR} the high
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products. In the case of @code{VEC_WIDEN_PLUS_LO_EXPR} the low
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products.
-
-@item VEC_WIDEN_MINUS_HI_EXPR
-@itemx VEC_WIDEN_MINUS_LO_EXPR
-These nodes represent widening vector subtraction of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The high/low
-elements of the second vector are subtracted from the high/low elements of the
-first. The result is a vector that contains half as many elements, of an
-integral type whose size is twice as wide. In the case of
-@code{VEC_WIDEN_MINUS_HI_EXPR} the high @code{N/2} elements of the second
-vector are subtracted from the high @code{N/2} of the first to produce the
-vector of @code{N/2} products. In the case of
-@code{VEC_WIDEN_MINUS_LO_EXPR} the low @code{N/2} elements of the second
-vector are subtracted from the low @code{N/2} of the first to produce the
-vector of @code{N/2} products.
-
@item VEC_UNPACK_HI_EXPR
@itemx VEC_UNPACK_LO_EXPR
These nodes represent unpacking of the high and low parts of the input vector,
diff --git a/gcc/expr.cc b/gcc/expr.cc
index 5d66c9f21f0ccd2eafb322eb9001f0dc873e35b4..b80385d51ba22172750d94535e04c82f75661255 100644
--- a/gcc/expr.cc
+++ b/gcc/expr.cc
@@ -9386,8 +9386,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
target, unsignedp);
return target;
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_MULT_EXPR:
/* If first operand is constant, swap them.
Thus the following special case checks need only
@@ -10165,10 +10163,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
return temp;
}
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/gimple-pretty-print.cc b/gcc/gimple-pretty-print.cc
index ebd87b20a0adc080c4a8f9429e75f49b96e72f9a..2a1a5b7f811ca341e8ee7e85a9701d3a37ff80bf 100644
--- a/gcc/gimple-pretty-print.cc
+++ b/gcc/gimple-pretty-print.cc
@@ -459,10 +459,6 @@ dump_binary_rhs (pretty_printer *buffer, const gassign *gs, int spc,
case VEC_PACK_FLOAT_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_SERIES_EXPR:
for (p = get_tree_code_name (code); *p; p++)
pp_character (buffer, TOUPPER (*p));
diff --git a/gcc/optabs-tree.cc b/gcc/optabs-tree.cc
index 8383fe820b080f6d66f83dcf3b77d3c9f869f4bc..2f5f93dc6624f86f6b5618cf6e7aa2b508053a64 100644
--- a/gcc/optabs-tree.cc
+++ b/gcc/optabs-tree.cc
@@ -190,22 +190,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
return (TYPE_UNSIGNED (type)
? vec_widen_ushiftl_lo_optab : vec_widen_sshiftl_lo_optab);
- case VEC_WIDEN_PLUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_lo_optab : vec_widen_saddl_lo_optab);
-
- case VEC_WIDEN_PLUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_hi_optab : vec_widen_saddl_hi_optab);
-
- case VEC_WIDEN_MINUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_lo_optab : vec_widen_ssubl_lo_optab);
-
- case VEC_WIDEN_MINUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_hi_optab : vec_widen_ssubl_hi_optab);
-
case VEC_UNPACK_HI_EXPR:
return (TYPE_UNSIGNED (type)
? vec_unpacku_hi_optab : vec_unpacks_hi_optab);
@@ -312,8 +296,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
'hi'/'lo' pair using codes such as VEC_WIDEN_MINUS_HI/LO.
Supported widening operations:
- WIDEN_MINUS_EXPR
- WIDEN_PLUS_EXPR
WIDEN_MULT_EXPR
WIDEN_LSHIFT_EXPR
@@ -345,12 +327,6 @@ supportable_half_widening_operation (enum tree_code code, tree vectype_out,
case WIDEN_LSHIFT_EXPR:
*code1 = LSHIFT_EXPR;
break;
- case WIDEN_MINUS_EXPR:
- *code1 = MINUS_EXPR;
- break;
- case WIDEN_PLUS_EXPR:
- *code1 = PLUS_EXPR;
- break;
case WIDEN_MULT_EXPR:
*code1 = MULT_EXPR;
break;
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index bfcb1425f7e2e46e3d525808adda11560041dd68..757c6c73e351c13bc6695699d9f449530546f70f 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -3951,8 +3951,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case PLUS_EXPR:
case MINUS_EXPR:
{
@@ -4073,10 +4071,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 043e1d5987a4c4b0159109dafb85a805ca828c1e..c0bebb7f4de36838341ed62389ad0e2b79f03034 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4288,8 +4288,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case REALIGN_LOAD_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case DOT_PROD_EXPR:
@@ -4298,10 +4296,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case WIDEN_MULT_MINUS_EXPR:
case WIDEN_LSHIFT_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-pretty-print.cc b/gcc/tree-pretty-print.cc
index bfabe9e76279d7c3383b684ed61cc92228de4500..0ca8802576656f098e60cb77fa4312d1375ff3f0 100644
--- a/gcc/tree-pretty-print.cc
+++ b/gcc/tree-pretty-print.cc
@@ -2846,8 +2846,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
break;
/* Binary arithmetic and logic expressions. */
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case MULT_EXPR:
@@ -3811,10 +3809,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
case VEC_SERIES_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
@@ -4332,12 +4326,6 @@ op_symbol_code (enum tree_code code)
case WIDEN_LSHIFT_EXPR:
return "w<<";
- case WIDEN_PLUS_EXPR:
- return "w+";
-
- case WIDEN_MINUS_EXPR:
- return "w-";
-
case POINTER_PLUS_EXPR:
return "+";
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index d20a10a1524164eef788ab4b88ba57c7a09c3387..98dd56ff022233ccead36a1f5a5e896e352f9f5b 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -136,8 +136,6 @@ vect_get_smallest_scalar_type (stmt_vec_info stmt_info, tree scalar_type)
|| gimple_assign_rhs_code (assign) == WIDEN_SUM_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_MULT_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_LSHIFT_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_PLUS_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_MINUS_EXPR
|| gimple_assign_rhs_code (assign) == FLOAT_EXPR)
{
tree rhs_type = TREE_TYPE (gimple_assign_rhs1 (assign));
@@ -3172,8 +3170,8 @@ vect_analyze_data_ref_accesses (vec_info *vinfo,
break;
/* Check that the DR_INITs are compile-time constants. */
- if (!tree_fits_shwi_p (DR_INIT (dra))
- || !tree_fits_shwi_p (DR_INIT (drb)))
+ if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST
+ || TREE_CODE (DR_INIT (drb)) != INTEGER_CST)
break;
/* Different .GOMP_SIMD_LANE calls still give the same lane,
@@ -3225,7 +3223,7 @@ vect_analyze_data_ref_accesses (vec_info *vinfo,
unsigned HOST_WIDE_INT step
= absu_hwi (tree_to_shwi (DR_STEP (dra)));
if (step != 0
- && step <= ((unsigned HOST_WIDE_INT)init_b - init_a))
+ && step <= (unsigned HOST_WIDE_INT)(init_b - init_a))
break;
}
}
diff --git a/gcc/tree-vect-generic.cc b/gcc/tree-vect-generic.cc
index 350129555a0c71c0896c4f1003163f3b3557c11b..066f05873118c2288c90604e6287c91ef9aed72b 100644
--- a/gcc/tree-vect-generic.cc
+++ b/gcc/tree-vect-generic.cc
@@ -2209,10 +2209,6 @@ expand_vector_operations_1 (gimple_stmt_iterator *gsi,
arguments, not the widened result. VEC_UNPACK_FLOAT_*_EXPR is
calculated in the same way above. */
if (code == WIDEN_SUM_EXPR
- || code == VEC_WIDEN_PLUS_HI_EXPR
- || code == VEC_WIDEN_PLUS_LO_EXPR
- || code == VEC_WIDEN_MINUS_HI_EXPR
- || code == VEC_WIDEN_MINUS_LO_EXPR
|| code == VEC_WIDEN_MULT_HI_EXPR
|| code == VEC_WIDEN_MULT_LO_EXPR
|| code == VEC_WIDEN_MULT_EVEN_EXPR
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 62ca28d725ed4ac8d7e4d493119e40772a0fbac6..9cd3989656c024b1d0394b2fcde6f6d774dff74e 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -559,21 +559,29 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else
+ rhs_code = gimple_call_combined_fn (stmt);
+
+ if (rhs_code.safe_as_tree_code () != code
+ && rhs_code.get_rep () != widened_code.get_rep ())
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = is_gimple_assign (stmt) ? gimple_assign_lhs (stmt):
+ gimple_call_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -586,7 +594,11 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op;
+ if (is_gimple_assign (stmt))
+ op = gimple_op (stmt, i + 1);
+ else
+ op = gimple_call_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1299,8 +1311,9 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
- false, 2, unprom, &half_type))
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ CFN_VEC_WIDEN_MINUS, false, 2, unprom,
+ &half_type))
return NULL;
vect_pattern_detected ("vect_recog_sad_pattern", last_stmt);
@@ -2337,9 +2350,10 @@ vect_recog_average_pattern (vec_info *vinfo,
internal_fn ifn = IFN_AVG_FLOOR;
vect_unpromoted_value unprom[3];
tree new_type;
+ enum optab_subtype subtype;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
- unprom, &new_type);
+ CFN_VEC_WIDEN_PLUS, false, 3,
+ unprom, &new_type, &subtype);
if (nops == 0)
return NULL;
if (nops == 3)
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 6fa0669fdfc8630842b3f9f32f4b4a253e79bb92..92b17e6d0ec18ce3d90290dba9efec5d1968264c 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4900,9 +4900,7 @@ vectorizable_conversion (vec_info *vinfo,
else
return false;
- bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
+ bool widen_arith = (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == IFN_VEC_WIDEN_PLUS
|| code == IFN_VEC_WIDEN_MINUS);
@@ -4954,8 +4952,6 @@ vectorizable_conversion (vec_info *vinfo,
{
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
|| code == IFN_VEC_WIDEN_PLUS
|| code == IFN_VEC_WIDEN_MINUS);
@@ -11986,7 +11982,7 @@ supportable_widening_operation (vec_info *vinfo,
class loop *vect_loop = NULL;
machine_mode vec_mode;
enum insn_code icode1, icode2;
- optab optab1, optab2;
+ optab optab1 = unknown_optab, optab2 = unknown_optab;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
code_helper c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
@@ -12080,16 +12076,6 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_WIDEN_LSHIFT_HI_EXPR;
break;
- case WIDEN_PLUS_EXPR:
- c1 = VEC_WIDEN_PLUS_LO_EXPR;
- c2 = VEC_WIDEN_PLUS_HI_EXPR;
- break;
-
- case WIDEN_MINUS_EXPR:
- c1 = VEC_WIDEN_MINUS_LO_EXPR;
- c2 = VEC_WIDEN_MINUS_HI_EXPR;
- break;
-
CASE_CONVERT:
c1 = VEC_UNPACK_LO_EXPR;
c2 = VEC_UNPACK_HI_EXPR;
diff --git a/gcc/tree.def b/gcc/tree.def
index 62650b6934b337c5d56e5393dc114173d72c9aa9..9b2dce3576440c445d3240b9ed937fe67c9a5992 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1383,8 +1383,6 @@ DEFTREECODE (WIDEN_MULT_MINUS_EXPR, "widen_mult_minus_expr", tcc_expression, 3)
the first argument from type t1 to type t2, and then shifting it
by the second argument. */
DEFTREECODE (WIDEN_LSHIFT_EXPR, "widen_lshift_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_PLUS_EXPR, "widen_plus_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_MINUS_EXPR, "widen_minus_expr", tcc_binary, 2)
/* Widening vector multiplication.
The two operands are vectors with N elements of size S. Multiplying the
@@ -1449,10 +1447,6 @@ DEFTREECODE (VEC_PACK_FLOAT_EXPR, "vec_pack_float_expr", tcc_binary, 2)
*/
DEFTREECODE (VEC_WIDEN_LSHIFT_HI_EXPR, "widen_lshift_hi_expr", tcc_binary, 2)
DEFTREECODE (VEC_WIDEN_LSHIFT_LO_EXPR, "widen_lshift_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_HI_EXPR, "widen_plus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_LO_EXPR, "widen_plus_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_HI_EXPR, "widen_minus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_LO_EXPR, "widen_minus_lo_expr", tcc_binary, 2)
/* PREDICT_EXPR. Specify hint for branch prediction. The
PREDICT_EXPR_PREDICTOR specify predictor and PREDICT_EXPR_OUTCOME the
--
2.17.1
^ permalink raw reply [flat|nested] 53+ messages in thread
* RE: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-06-30 13:20 ` Joel Hutton
@ 2022-07-12 12:32 ` Richard Biener
2023-03-17 10:14 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2022-07-12 12:32 UTC (permalink / raw)
To: Joel Hutton; +Cc: Richard Sandiford, gcc-patches, Andre Simoes Dias Vieira
On Thu, 30 Jun 2022, Joel Hutton wrote:
> > We can go with a private vect_gimple_build function until we sort out the API
> > issue to unblock Tamar (I'll reply to Richards reply with further thoughts on
> > this)
> >
>
> Done.
>
> > > Similarly are you ok with the use of gimple_extract_op? I would lean
> > towards using it as it is cleaner, but I don't have strong feelings.
> >
> > I don't like using gimple_extract_op here, I think I outlined a variant that is
> > even shorter.
> >
>
> Done.
>
> Updated patches attached, bootstrapped and regression tested on aarch64.
>
> Tomorrow is my last working day at Arm, so it will likely be Andre that commits this/addresses any further comments.
First sorry for the (repeated) delays.
In the first patch I still see ECF_WIDEN, I don't like that, we
use things like associative_binary_fn_p so for widening internal
functions similar predicates should be used.
In the second patch you add vec_widen_{add,sub} optabs
+OPTAB_CD(vec_widen_add_optab, "add$a$b3")
+OPTAB_CD(vec_widen_sub_optab, "sub$a$b3")
but a) the names are that of regular adds which is at least confusing
(if not wrong), b) there's no documentation for them in md.texi which,
c) doesn't explain why they are necessary when we have
vec_widen_[su]{add,sub}l_optab
+ internal_fn ifn = as_internal_fn (code.safe_as_fn_code ());
asks for safe_as_internal_fn () (just complete the API, also with
safe_as_builtin_fn)
+ internal_fn lo, hi;
+ lookup_multi_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
in fact this probably shows that the guarding condition should
be if (code.is_internal_fn ()) instead of if (code.is_fn_code ()).
+ optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
this shows the two lookup_ APIs are inconsistent in having two vs. one
output, please make them consistent. I'd say give
lookup_multi_internal_fn a enum { LO, HI } argument, returning the
result. Given VEC_WIDEN_MULT has HI, LO, EVEN and ODD variants
that sounds more future proof.
The internal_fn stuff could probably get a 2nd eye from Richard.
In the third patch I see unrelated and wrong changes like
/* Check that the DR_INITs are compile-time constants. */
- if (!tree_fits_shwi_p (DR_INIT (dra))
- || !tree_fits_shwi_p (DR_INIT (drb)))
+ if (TREE_CODE (DR_INIT (dra)) != INTEGER_CST
+ || TREE_CODE (DR_INIT (drb)) != INTEGER_CST)
break;
please strip the patch down to relevant changes.
- tree op = gimple_op (assign, i + 1);
+ tree op;
+ if (is_gimple_assign (stmt))
+ op = gimple_op (stmt, i + 1);
+ else
+ op = gimple_call_arg (stmt, i);
somebody added gimple_arg which can be used here doing
op = gimple_arg (stmt, i);
+ tree lhs = is_gimple_assign (stmt) ? gimple_assign_lhs (stmt):
+ gimple_call_lhs (stmt);
tree lhs = gimple_get_lhs (stmt);
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else
+ rhs_code = gimple_call_combined_fn (stmt);
+
+ if (rhs_code.safe_as_tree_code () != code
+ && rhs_code.get_rep () != widened_code.get_rep ())
return 0;
that's probably better refactored as
if (is_gimple_assign (stmt))
{
if (code check)
return 0;
}
else if (is_gimple_call (..))
{
..
}
else
return 0;
otherwise the last patch looks reasonable.
Richard.
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2022-07-12 12:32 ` Richard Biener
@ 2023-03-17 10:14 ` Andre Vieira (lists)
2023-03-17 11:52 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-03-17 10:14 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Sandiford, gcc-patches
Hi Richard,
I'm only picking this up now. Just going through your earlier comments
and stuff and I noticed we didn't address the situation with the
gimple::build. Do you want me to add overloaded static member functions
to cover all gimple_build_* functions, or just create one to replace
vect_gimple_build and we create them as needed? It's more work but I
think adding them all would be better. I'd even argue that it would be
nice to replace the old ones with the new ones, but I can imagine you
might not want that as it makes backporting and the likes a bit annoying...
Let me know what you prefer, I'll go work on your latest comments too.
Cheers,
Andre
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2023-03-17 10:14 ` Andre Vieira (lists)
@ 2023-03-17 11:52 ` Richard Biener
2023-04-20 13:23 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-03-17 11:52 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Sandiford, gcc-patches
On Fri, 17 Mar 2023, Andre Vieira (lists) wrote:
> Hi Richard,
>
> I'm only picking this up now. Just going through your earlier comments and
> stuff and I noticed we didn't address the situation with the gimple::build. Do
> you want me to add overloaded static member functions to cover all
> gimple_build_* functions, or just create one to replace vect_gimple_build and
> we create them as needed? It's more work but I think adding them all would be
> better. I'd even argue that it would be nice to replace the old ones with the
> new ones, but I can imagine you might not want that as it makes backporting
> and the likes a bit annoying...
>
> Let me know what you prefer, I'll go work on your latest comments too.
I think the series was resolved and I approved it. As for
vect_gimple_build the better way forward would be to use
gimple_build () as existing but add a vect_finish_stmt_* handling
a gimple_seq.
Richard.
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2023-03-17 11:52 ` Richard Biener
@ 2023-04-20 13:23 ` Andre Vieira (lists)
2023-04-24 11:57 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-04-20 13:23 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]
Rebased all three patches and made some small changes to the second one:
- removed sub and abd optabs from commutative_optab_p, I suspect this
was a copy paste mistake,
- removed what I believe to be a superfluous switch case in vectorizable
conversion, the one that was here:
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn (code.as_fn_code ());
+ int ecf_flags = internal_fn_flags (ifn);
+ gcc_assert (ecf_flags & ECF_MULTI);
+
+ switch (code.as_fn_code ())
+ {
+ case CFN_VEC_WIDEN_PLUS:
+ break;
+ case CFN_VEC_WIDEN_MINUS:
+ break;
+ case CFN_LAST:
+ default:
+ return false;
+ }
+
+ internal_fn lo, hi;
+ lookup_multi_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
}
I don't think we need to check they are a specfic fn code, as we look-up
optabs and if they succeed then surely we can vectorize?
OK for trunk?
Kind regards,
Andre
[-- Attachment #2: ifn0.patch --]
[-- Type: text/plain, Size: 21121 bytes --]
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 8802141cd6edb298866025b8a55843eae1f0eb17..68dfba266d679c9738a3d5d70551a91cbdafcf66 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -25,6 +25,8 @@ along with GCC; see the file COPYING3. If not see
#include "rtl.h"
#include "tree.h"
#include "gimple.h"
+#include "gimple-iterator.h"
+#include "gimple-fold.h"
#include "ssa.h"
#include "expmed.h"
#include "optabs-tree.h"
@@ -1391,7 +1393,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
- tree_code orig_code, tree_code wide_code,
+ tree_code orig_code, code_helper wide_code,
bool shift_p, const char *name)
{
gimple *last_stmt = last_stmt_info->stmt;
@@ -1434,7 +1436,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
vecctype = get_vectype_for_scalar_type (vinfo, ctype);
}
- enum tree_code dummy_code;
+ code_helper dummy_code;
int dummy_int;
auto_vec<tree> dummy_vec;
if (!vectype
@@ -1455,8 +1457,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
2, oprnd, half_type, unprom, vectype);
tree var = vect_recog_temp_ssa_var (itype, NULL);
- gimple *pattern_stmt = gimple_build_assign (var, wide_code,
- oprnd[0], oprnd[1]);
+ gimple *pattern_stmt = vect_gimple_build (var, wide_code, oprnd[0], oprnd[1]);
if (vecctype != vecitype)
pattern_stmt = vect_convert_output (vinfo, last_stmt_info, ctype,
@@ -6406,3 +6407,28 @@ vect_pattern_recog (vec_info *vinfo)
/* After this no more add_stmt calls are allowed. */
vinfo->stmt_vec_info_ro = true;
}
+
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple *
+vect_gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
+{
+ if (op0 == NULL_TREE)
+ return NULL;
+ if (ch.is_tree_code ())
+ return op1 == NULL_TREE ? gimple_build_assign (lhs, ch.safe_as_tree_code (),
+ op0) :
+ gimple_build_assign (lhs, ch.safe_as_tree_code (),
+ op0, op1);
+ else
+ {
+ internal_fn fn = as_internal_fn (ch.safe_as_fn_code ());
+ gimple* stmt;
+ if (op1 == NULL_TREE)
+ stmt = gimple_build_call_internal (fn, 1, op0);
+ else
+ stmt = gimple_build_call_internal (fn, 2, op0, op1);
+ gimple_call_set_lhs (stmt, lhs);
+ return stmt;
+ }
+}
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 6b7dbfd4a231baec24e740ffe0ce0b0bf7a1de6b..715ec2e30a4de620b8a5076c0e7f2f7fd1b0654e 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -4768,7 +4768,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
STMT_INFO is the original scalar stmt that we are vectorizing. */
static gimple *
-vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
+vect_gen_widened_results_half (vec_info *vinfo, code_helper ch,
tree vec_oprnd0, tree vec_oprnd1, int op_type,
tree vec_dest, gimple_stmt_iterator *gsi,
stmt_vec_info stmt_info)
@@ -4777,12 +4777,11 @@ vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
tree new_temp;
/* Generate half of the widened result: */
- gcc_assert (op_type == TREE_CODE_LENGTH (code));
if (op_type != binary_op)
vec_oprnd1 = NULL;
- new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0, vec_oprnd1);
+ new_stmt = vect_gimple_build (vec_dest, ch, vec_oprnd0, vec_oprnd1);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
return new_stmt;
@@ -4861,8 +4860,8 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
- enum tree_code code2, int op_type)
+ code_helper ch1,
+ code_helper ch2, int op_type)
{
int i;
tree vop0, vop1, new_tmp1, new_tmp2;
@@ -4878,10 +4877,10 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vop1 = NULL_TREE;
/* Generate the two halves of promotion operation. */
- new_stmt1 = vect_gen_widened_results_half (vinfo, code1, vop0, vop1,
+ new_stmt1 = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
- new_stmt2 = vect_gen_widened_results_half (vinfo, code2, vop0, vop1,
+ new_stmt2 = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
if (is_gimple_call (new_stmt1))
@@ -4978,8 +4977,9 @@ vectorizable_conversion (vec_info *vinfo,
tree scalar_dest;
tree op0, op1 = NULL_TREE;
loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
- enum tree_code code, code1 = ERROR_MARK, code2 = ERROR_MARK;
- enum tree_code codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
+ tree_code tc1;
+ code_helper code, code1, code2;
+ code_helper codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
tree new_temp;
enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type};
int ndts = 2;
@@ -5008,31 +5008,43 @@ vectorizable_conversion (vec_info *vinfo,
&& ! vec_stmt)
return false;
- gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
- if (!stmt)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return false;
- if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
+ if (gimple_get_lhs (stmt) == NULL_TREE
+ || TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- code = gimple_assign_rhs_code (stmt);
- if (!CONVERT_EXPR_CODE_P (code)
- && code != FIX_TRUNC_EXPR
- && code != FLOAT_EXPR
- && code != WIDEN_PLUS_EXPR
- && code != WIDEN_MINUS_EXPR
- && code != WIDEN_MULT_EXPR
- && code != WIDEN_LSHIFT_EXPR)
+ if (TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
+ return false;
+
+ if (is_gimple_assign (stmt))
+ {
+ code = gimple_assign_rhs_code (stmt);
+ op_type = TREE_CODE_LENGTH (code.safe_as_tree_code ());
+ }
+ else if (gimple_call_internal_p (stmt))
+ {
+ code = gimple_call_internal_fn (stmt);
+ op_type = gimple_call_num_args (stmt);
+ }
+ else
return false;
bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
- op_type = TREE_CODE_LENGTH (code);
+ || code == WIDEN_MINUS_EXPR
+ || code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR);
+
+ if (!widen_arith
+ && !CONVERT_EXPR_CODE_P (code)
+ && code != FIX_TRUNC_EXPR
+ && code != FLOAT_EXPR)
+ return false;
/* Check types of lhs and rhs. */
- scalar_dest = gimple_assign_lhs (stmt);
+ scalar_dest = gimple_get_lhs (stmt);
lhs_type = TREE_TYPE (scalar_dest);
vectype_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -5070,10 +5082,14 @@ vectorizable_conversion (vec_info *vinfo,
if (op_type == binary_op)
{
- gcc_assert (code == WIDEN_MULT_EXPR || code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR || code == WIDEN_MINUS_EXPR);
+ gcc_assert (code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR
+ || code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR);
+
- op1 = gimple_assign_rhs2 (stmt);
+ op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
+ gimple_call_arg (stmt, 0);
tree vectype1_in;
if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
&op1, &slp_op1, &dt[1], &vectype1_in))
@@ -5157,8 +5173,12 @@ vectorizable_conversion (vec_info *vinfo,
&& code != FLOAT_EXPR
&& !CONVERT_EXPR_CODE_P (code))
return false;
- if (supportable_convert_operation (code, vectype_out, vectype_in, &code1))
+ if (supportable_convert_operation (code.safe_as_tree_code (), vectype_out,
+ vectype_in, &tc1))
+ {
+ code1 = tc1;
break;
+ }
/* FALLTHRU */
unsupported:
if (dump_enabled_p ())
@@ -5169,9 +5189,11 @@ vectorizable_conversion (vec_info *vinfo,
case WIDEN:
if (known_eq (nunits_in, nunits_out))
{
- if (!supportable_half_widening_operation (code, vectype_out,
- vectype_in, &code1))
+ if (!supportable_half_widening_operation (code.safe_as_tree_code (),
+ vectype_out, vectype_in,
+ &tc1))
goto unsupported;
+ code1 = tc1;
gcc_assert (!(multi_step_cvt && op_type == binary_op));
break;
}
@@ -5205,14 +5227,17 @@ vectorizable_conversion (vec_info *vinfo,
if (GET_MODE_SIZE (rhs_mode) == fltsz)
{
- if (!supportable_convert_operation (code, vectype_out,
- cvt_type, &codecvt1))
+ tc1 = ERROR_MARK;
+ if (!supportable_convert_operation (code.safe_as_tree_code (),
+ vectype_out,
+ cvt_type, &tc1))
goto unsupported;
+ codecvt1 = tc1;
}
- else if (!supportable_widening_operation (vinfo, code, stmt_info,
- vectype_out, cvt_type,
- &codecvt1, &codecvt2,
- &multi_step_cvt,
+ else if (!supportable_widening_operation (vinfo, code,
+ stmt_info, vectype_out,
+ cvt_type, &codecvt1,
+ &codecvt2, &multi_step_cvt,
&interm_types))
continue;
else
@@ -5220,8 +5245,9 @@ vectorizable_conversion (vec_info *vinfo,
if (supportable_widening_operation (vinfo, NOP_EXPR, stmt_info,
cvt_type,
- vectype_in, &code1, &code2,
- &multi_step_cvt, &interm_types))
+ vectype_in, &code1,
+ &code2, &multi_step_cvt,
+ &interm_types))
{
found_mode = true;
break;
@@ -5243,10 +5269,15 @@ vectorizable_conversion (vec_info *vinfo,
case NARROW:
gcc_assert (op_type == unary_op);
- if (supportable_narrowing_operation (code, vectype_out, vectype_in,
- &code1, &multi_step_cvt,
+ if (supportable_narrowing_operation (code.safe_as_tree_code (),
+ vectype_out,
+ vectype_in,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
if (code != FIX_TRUNC_EXPR
|| GET_MODE_SIZE (lhs_mode) >= GET_MODE_SIZE (rhs_mode))
@@ -5257,13 +5288,18 @@ vectorizable_conversion (vec_info *vinfo,
cvt_type = get_same_sized_vectype (cvt_type, vectype_in);
if (cvt_type == NULL_TREE)
goto unsupported;
- if (!supportable_convert_operation (code, cvt_type, vectype_in,
- &codecvt1))
+ if (!supportable_convert_operation (code.safe_as_tree_code (), cvt_type,
+ vectype_in,
+ &tc1))
goto unsupported;
+ codecvt1 = tc1;
if (supportable_narrowing_operation (NOP_EXPR, vectype_out, cvt_type,
- &code1, &multi_step_cvt,
+ &tc1, &multi_step_cvt,
&interm_types))
- break;
+ {
+ code1 = tc1;
+ break;
+ }
goto unsupported;
default:
@@ -5377,8 +5413,10 @@ vectorizable_conversion (vec_info *vinfo,
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
/* Arguments are ready, create the new vector stmt. */
- gcc_assert (TREE_CODE_LENGTH (code1) == unary_op);
- gassign *new_stmt = gimple_build_assign (vec_dest, code1, vop0);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) code1) == unary_op);
+ gassign *new_stmt = gimple_build_assign (vec_dest,
+ code1.safe_as_tree_code (),
+ vop0);
new_temp = make_ssa_name (vec_dest, new_stmt);
gimple_assign_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
@@ -5410,7 +5448,7 @@ vectorizable_conversion (vec_info *vinfo,
for (i = multi_step_cvt; i >= 0; i--)
{
tree this_dest = vec_dsts[i];
- enum tree_code c1 = code1, c2 = code2;
+ code_helper c1 = code1, c2 = code2;
if (i == 0 && codecvt2 != ERROR_MARK)
{
c1 = codecvt1;
@@ -5420,7 +5458,8 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_half_widening_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
this_dest, gsi,
- c1, op_type);
+ c1.safe_as_tree_code (),
+ op_type);
else
vect_create_vectorized_promotion_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
@@ -5433,9 +5472,11 @@ vectorizable_conversion (vec_info *vinfo,
gimple *new_stmt;
if (cvt_type)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH ((tree_code) codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, codecvt1, vop0);
+ new_stmt = gimple_build_assign (new_temp,
+ codecvt1.safe_as_tree_code (),
+ vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -5459,10 +5500,12 @@ vectorizable_conversion (vec_info *vinfo,
if (cvt_type)
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
+ gcc_assert (TREE_CODE_LENGTH (((tree_code) codecvt1)) == unary_op);
new_temp = make_ssa_name (vec_dest);
gassign *new_stmt
- = gimple_build_assign (new_temp, codecvt1, vop0);
+ = gimple_build_assign (new_temp,
+ codecvt1.safe_as_tree_code (),
+ vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
vec_oprnds0[i] = new_temp;
}
@@ -5470,7 +5513,8 @@ vectorizable_conversion (vec_info *vinfo,
vect_create_vectorized_demotion_stmts (vinfo, &vec_oprnds0,
multi_step_cvt,
stmt_info, vec_dsts, gsi,
- slp_node, code1);
+ slp_node,
+ code1.safe_as_tree_code ());
break;
}
if (!slp_node)
@@ -12151,9 +12195,11 @@ vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype)
bool
supportable_widening_operation (vec_info *vinfo,
- enum tree_code code, stmt_vec_info stmt_info,
+ code_helper code,
+ stmt_vec_info stmt_info,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, enum tree_code *code2,
+ code_helper *code1,
+ code_helper *code2,
int *multi_step_cvt,
vec<tree> *interm_types)
{
@@ -12164,7 +12210,7 @@ supportable_widening_operation (vec_info *vinfo,
optab optab1, optab2;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
- enum tree_code c1, c2;
+ code_helper c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
int i;
tree prev_type, intermediate_type;
machine_mode intermediate_mode, prev_mode;
@@ -12174,7 +12220,7 @@ supportable_widening_operation (vec_info *vinfo,
if (loop_info)
vect_loop = LOOP_VINFO_LOOP (loop_info);
- switch (code)
+ switch (code.safe_as_tree_code ())
{
case WIDEN_MULT_EXPR:
/* The result of a vectorized widening operation usually requires
@@ -12215,8 +12261,9 @@ supportable_widening_operation (vec_info *vinfo,
&& !nested_in_vect_loop_p (vect_loop, stmt_info)
&& supportable_widening_operation (vinfo, VEC_WIDEN_MULT_EVEN_EXPR,
stmt_info, vectype_out,
- vectype_in, code1, code2,
- multi_step_cvt, interm_types))
+ vectype_in, code1,
+ code2, multi_step_cvt,
+ interm_types))
{
/* Elements in a vector with vect_used_by_reduction property cannot
be reordered if the use chain with this property does not have the
@@ -12279,6 +12326,9 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_UNPACK_FIX_TRUNC_HI_EXPR;
break;
+ case MAX_TREE_CODES:
+ break;
+
default:
gcc_unreachable ();
}
@@ -12289,10 +12339,12 @@ supportable_widening_operation (vec_info *vinfo,
if (code == FIX_TRUNC_EXPR)
{
/* The signedness is determined from output operand. */
- optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
- optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype_out,
+ optab_default);
}
- else if (CONVERT_EXPR_CODE_P (code)
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
&& VECTOR_BOOLEAN_TYPE_P (wide_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
&& TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
@@ -12305,8 +12357,10 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype,
+ optab_default);
}
if (!optab1 || !optab2)
@@ -12317,8 +12371,12 @@ supportable_widening_operation (vec_info *vinfo,
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- *code1 = c1;
- *code2 = c2;
+ if (code.is_tree_code ())
+ {
+ *code1 = c1;
+ *code2 = c2;
+ }
+
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
@@ -12339,7 +12397,7 @@ supportable_widening_operation (vec_info *vinfo,
prev_type = vectype;
prev_mode = vec_mode;
- if (!CONVERT_EXPR_CODE_P (code))
+ if (!CONVERT_EXPR_CODE_P ((tree_code) code))
return false;
/* We assume here that there will not be more than MAX_INTERM_CVT_STEPS
@@ -12379,8 +12437,12 @@ supportable_widening_operation (vec_info *vinfo,
}
else
{
- optab3 = optab_for_tree_code (c1, intermediate_type, optab_default);
- optab4 = optab_for_tree_code (c2, intermediate_type, optab_default);
+ optab3 = optab_for_tree_code (c1.safe_as_tree_code (),
+ intermediate_type,
+ optab_default);
+ optab4 = optab_for_tree_code (c2.safe_as_tree_code (),
+ intermediate_type,
+ optab_default);
}
if (!optab3 || !optab4
@@ -12439,7 +12501,7 @@ supportable_widening_operation (vec_info *vinfo,
bool
supportable_narrowing_operation (enum tree_code code,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, int *multi_step_cvt,
+ tree_code *code1, int *multi_step_cvt,
vec<tree> *interm_types)
{
machine_mode vec_mode;
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 9cf2fb23fe397b467d89aa7cc5ebeaa293ed4cce..d241eba6ef3302225bbe37b374baa11e6472c280 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2139,13 +2139,12 @@ extern bool vect_is_simple_use (vec_info *, stmt_vec_info, slp_tree,
enum vect_def_type *,
tree *, stmt_vec_info * = NULL);
extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
-extern bool supportable_widening_operation (vec_info *,
- enum tree_code, stmt_vec_info,
- tree, tree, enum tree_code *,
- enum tree_code *, int *,
- vec<tree> *);
+extern bool supportable_widening_operation (vec_info*, code_helper,
+ stmt_vec_info, tree, tree,
+ code_helper*, code_helper*,
+ int*, vec<tree> *);
extern bool supportable_narrowing_operation (enum tree_code, tree, tree,
- enum tree_code *, int *,
+ tree_code *, int *,
vec<tree> *);
extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
@@ -2583,4 +2582,7 @@ vect_is_integer_truncation (stmt_vec_info stmt_info)
&& TYPE_PRECISION (lhs_type) < TYPE_PRECISION (rhs_type));
}
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple * vect_gimple_build (tree, code_helper, tree, tree);
#endif /* GCC_TREE_VECTORIZER_H */
diff --git a/gcc/tree.h b/gcc/tree.h
index abcdb5638d49aea4ccc46efa8e540b1fa78aa27a..a250a80e0321241e1158086acb2dd837d5827e10 100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -93,6 +93,8 @@ public:
bool is_internal_fn () const;
bool is_builtin_fn () const;
int get_rep () const { return rep; }
+ enum tree_code safe_as_tree_code () const;
+ combined_fn safe_as_fn_code () const;
bool operator== (const code_helper &other) { return rep == other.rep; }
bool operator!= (const code_helper &other) { return rep != other.rep; }
bool operator== (tree_code c) { return rep == code_helper (c).rep; }
@@ -102,6 +104,17 @@ private:
int rep;
};
+inline enum tree_code
+code_helper::safe_as_tree_code () const
+{
+ return is_tree_code () ? (tree_code)* this : MAX_TREE_CODES;
+}
+
+inline combined_fn
+code_helper::safe_as_fn_code () const {
+ return is_fn_code () ? (combined_fn) *this : CFN_LAST;
+}
+
inline code_helper::operator internal_fn () const
{
return as_internal_fn (combined_fn (*this));
[-- Attachment #3: ifn1.patch --]
[-- Type: text/plain, Size: 18605 bytes --]
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 6e81dc05e0e0714256759b0594816df451415a2d..e4d815cd577d266d2bccf6fb68d62aac91a8b4cf 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -17,6 +17,7 @@ You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3. If not see
<http://www.gnu.org/licenses/>. */
+#define INCLUDE_MAP
#include "config.h"
#include "system.h"
#include "coretypes.h"
@@ -70,6 +71,26 @@ const int internal_fn_flags_array[] = {
0
};
+const enum internal_fn internal_fn_hilo_keys_array[] = {
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ IFN_##NAME##_LO, \
+ IFN_##NAME##_HI,
+#include "internal-fn.def"
+ IFN_LAST
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+};
+
+const optab internal_fn_hilo_values_array[] = {
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ SOPTAB##_lo_optab, UOPTAB##_lo_optab, \
+ SOPTAB##_hi_optab, UOPTAB##_hi_optab,
+#include "internal-fn.def"
+ unknown_optab, unknown_optab
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+};
+
/* Return the internal function called NAME, or IFN_LAST if there's
no such function. */
@@ -90,6 +111,61 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+static int
+ifn_cmp (const void *a_, const void *b_)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ auto *a = (const std::pair<ifn_pair, optab> *)a_;
+ auto *b = (const std::pair<ifn_pair, optab> *)b_;
+ return (int) (a->first.first) - (b->first.first);
+}
+
+/* Return the optab belonging to the given internal function NAME for the given
+ SIGN or unknown_optab. */
+
+optab
+lookup_hilo_ifn_optab (enum internal_fn fn, unsigned sign)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ typedef auto_vec <std::pair<ifn_pair, optab>>fn_to_optab_map_type;
+ static fn_to_optab_map_type *fn_to_optab_map;
+
+ if (!fn_to_optab_map)
+ {
+ unsigned num
+ = sizeof (internal_fn_hilo_keys_array) / sizeof (enum internal_fn);
+ fn_to_optab_map = new fn_to_optab_map_type ();
+ for (unsigned int i = 0; i < num - 1; ++i)
+ {
+ enum internal_fn fn = internal_fn_hilo_keys_array[i];
+ optab v1 = internal_fn_hilo_values_array[2*i];
+ optab v2 = internal_fn_hilo_values_array[2*i + 1];
+ ifn_pair key1 (fn, 0);
+ fn_to_optab_map->safe_push ({key1, v1});
+ ifn_pair key2 (fn, 1);
+ fn_to_optab_map->safe_push ({key2, v2});
+ }
+ fn_to_optab_map->qsort (ifn_cmp);
+ }
+
+ ifn_pair new_pair (fn, sign ? 1 : 0);
+ optab tmp;
+ std::pair<ifn_pair,optab> pair_wrap (new_pair, tmp);
+ auto entry = fn_to_optab_map->bsearch (&pair_wrap, ifn_cmp);
+ return entry != fn_to_optab_map->end () ? entry->second : unknown_optab;
+}
+
+extern void
+lookup_hilo_internal_fn (enum internal_fn ifn, enum internal_fn *lo,
+ enum internal_fn *hi)
+{
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -3970,6 +4046,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -4043,6 +4122,42 @@ first_commutative_argument (internal_fn fn)
}
}
+/* Return true if FN has a wider output type than its argument types. */
+
+bool
+widening_fn_p (internal_fn fn)
+{
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_MINUS:
+ return true;
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if FN decomposes to _hi and _lo IFN. If true this should also
+ be a widening function. */
+
+bool
+decomposes_to_hilo_fn_p (internal_fn fn)
+{
+ if (!widening_fn_p (fn))
+ return false;
+
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_MINUS:
+ return true;
+
+ default:
+ return false;
+ }
+}
+
/* Return true if IFN_SET_EDOM is supported. */
bool
@@ -4055,6 +4170,32 @@ set_edom_supported_p (void)
#endif
}
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(CODE, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ static void \
+ expand_##CODE (internal_fn, gcall *) \
+ { \
+ gcc_unreachable (); \
+ } \
+ static void \
+ expand_##CODE##_LO (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_lo##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_lo##_optab); \
+ } \
+ static void \
+ expand_##CODE##_HI (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_hi##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_hi##_optab); \
+ }
+
#define DEF_INTERNAL_OPTAB_FN(CODE, FLAGS, OPTAB, TYPE) \
static void \
expand_##CODE (internal_fn fn, gcall *stmt) \
@@ -4071,6 +4212,7 @@ set_edom_supported_p (void)
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_HILO_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 7fe742c2ae713e7152ab05cfdfba86e4e0aa3456..347ed667d92620e0ee3ea15c58ecac6c242ebe73 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -85,6 +85,13 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_OPTAB_HILO_FN is like DEF_INTERNAL_OPTAB_FN except it
+ provides convenience wrappers for defining conversions that require a
+ hi/lo split, like widening and narrowing operations. Each definition
+ for <NAME> will require an optab named <OPTAB> and two other optabs that
+ you specify for signed and unsigned.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -123,6 +130,14 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, unknown, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, unknown, TYPE)
+#endif
+
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -315,6 +330,14 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_OPTAB_HILO_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_NOTHROW,
+ vec_widen_add, vec_widen_saddl, vec_widen_uaddl,
+ binary)
+DEF_INTERNAL_OPTAB_HILO_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_NOTHROW,
+ vec_widen_sub, vec_widen_ssubl, vec_widen_usubl,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 08922ed4254898f5fffca3f33973e96ed9ce772f..6a5f8762e872ad2ef64ce2986a678e3b40622d81 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,9 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern optab lookup_hilo_ifn_optab (enum internal_fn, unsigned);
+extern void lookup_hilo_internal_fn (enum internal_fn, enum internal_fn *,
+ enum internal_fn *);
/* Return the ECF_* flags for function FN. */
@@ -210,6 +217,8 @@ extern bool commutative_binary_fn_p (internal_fn);
extern bool commutative_ternary_fn_p (internal_fn);
extern int first_commutative_argument (internal_fn);
extern bool associative_binary_fn_p (internal_fn);
+extern bool widening_fn_p (internal_fn);
+extern bool decomposes_to_hilo_fn_p (internal_fn);
extern bool set_edom_supported_p (void);
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index c8e39c82d57a7d726e7da33d247b80f32ec9236c..d4dd7ee3d34d01c32ab432ae4e4ce9e4b522b2f7 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,12 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_add_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 695f5911b300c9ca5737de9be809fa01aabe5e01..e064189103b3be70644468d11f3c91ac45ffe0d0 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -78,6 +78,8 @@ OPTAB_CD(smsub_widen_optab, "msub$b$a4")
OPTAB_CD(umsub_widen_optab, "umsub$b$a4")
OPTAB_CD(ssmsub_widen_optab, "ssmsub$b$a4")
OPTAB_CD(usmsub_widen_optab, "usmsub$a$b4")
+OPTAB_CD(vec_widen_add_optab, "add$a$b3")
+OPTAB_CD(vec_widen_sub_optab, "sub$a$b3")
OPTAB_CD(vec_load_lanes_optab, "vec_load_lanes$a$b")
OPTAB_CD(vec_store_lanes_optab, "vec_store_lanes$a$b")
OPTAB_CD(vec_mask_load_lanes_optab, "vec_mask_load_lanes$a$b")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 68dfba266d679c9738a3d5d70551a91cbdafcf66..1a514461b2ca416f45a5fa9abe417980d33ef4df 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1394,14 +1394,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1467,6 +1469,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1480,26 +1496,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_popcount_pattern
@@ -6067,6 +6087,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 715ec2e30a4de620b8a5076c0e7f2f7fd1b0654e..f4806073f48d4dedea3ac9bd855792b152d78919 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5035,7 +5035,9 @@ vectorizable_conversion (vec_info *vinfo,
bool widen_arith = (code == WIDEN_PLUS_EXPR
|| code == WIDEN_MINUS_EXPR
|| code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
+ || code == WIDEN_LSHIFT_EXPR
+ || code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -5085,7 +5087,9 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
+ || code == WIDEN_MINUS_EXPR
+ || code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
@@ -12355,14 +12359,50 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
- {
- optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype,
- optab_default);
- optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype,
- optab_default);
+
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn (code.safe_as_fn_code ());
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ internal_fn lo, hi;
+ lookup_hilo_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = lookup_hilo_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_hilo_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
}
+ if (code.is_tree_code ())
+ {
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype_out,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype_out,
+ optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1.safe_as_tree_code (), vectype,
+ optab_default);
+ optab2 = optab_for_tree_code (c2.safe_as_tree_code (), vectype,
+ optab_default);
+ }
+ }
+
if (!optab1 || !optab2)
return false;
[-- Attachment #4: ifn2.patch --]
[-- Type: text/plain, Size: 19234 bytes --]
diff --git a/gcc/cfgexpand.cc b/gcc/cfgexpand.cc
index 1a1b26b1c6c23ce273bcd08dc9a973f777174007..25b1558dcb941ea491a19aeeb2cd8f4d2dbdf7c6 100644
--- a/gcc/cfgexpand.cc
+++ b/gcc/cfgexpand.cc
@@ -5365,10 +5365,6 @@ expand_debug_expr (tree exp)
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_PERM_EXPR:
case VEC_DUPLICATE_EXPR:
case VEC_SERIES_EXPR:
@@ -5405,8 +5401,6 @@ expand_debug_expr (tree exp)
case WIDEN_MULT_EXPR:
case WIDEN_MULT_PLUS_EXPR:
case WIDEN_MULT_MINUS_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
if (SCALAR_INT_MODE_P (GET_MODE (op0))
&& SCALAR_INT_MODE_P (mode))
{
@@ -5419,10 +5413,6 @@ expand_debug_expr (tree exp)
op1 = simplify_gen_unary (ZERO_EXTEND, mode, op1, inner_mode);
else
op1 = simplify_gen_unary (SIGN_EXTEND, mode, op1, inner_mode);
- if (TREE_CODE (exp) == WIDEN_PLUS_EXPR)
- return simplify_gen_binary (PLUS, mode, op0, op1);
- else if (TREE_CODE (exp) == WIDEN_MINUS_EXPR)
- return simplify_gen_binary (MINUS, mode, op0, op1);
op0 = simplify_gen_binary (MULT, mode, op0, op1);
if (TREE_CODE (exp) == WIDEN_MULT_EXPR)
return op0;
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index 2c14b7abce2db0a3da0a21e916907947cb56a265..3816abaaf4d364d604a44942317f96f3f303e5b6 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,10 +1811,6 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1861,33 +1857,6 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
-@item VEC_WIDEN_PLUS_HI_EXPR
-@itemx VEC_WIDEN_PLUS_LO_EXPR
-These nodes represent widening vector addition of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The result
-is a vector that contains half as many elements, of an integral type whose size
-is twice as wide. In the case of @code{VEC_WIDEN_PLUS_HI_EXPR} the high
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products. In the case of @code{VEC_WIDEN_PLUS_LO_EXPR} the low
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products.
-
-@item VEC_WIDEN_MINUS_HI_EXPR
-@itemx VEC_WIDEN_MINUS_LO_EXPR
-These nodes represent widening vector subtraction of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The high/low
-elements of the second vector are subtracted from the high/low elements of the
-first. The result is a vector that contains half as many elements, of an
-integral type whose size is twice as wide. In the case of
-@code{VEC_WIDEN_MINUS_HI_EXPR} the high @code{N/2} elements of the second
-vector are subtracted from the high @code{N/2} of the first to produce the
-vector of @code{N/2} products. In the case of
-@code{VEC_WIDEN_MINUS_LO_EXPR} the low @code{N/2} elements of the second
-vector are subtracted from the low @code{N/2} of the first to produce the
-vector of @code{N/2} products.
-
@item VEC_UNPACK_HI_EXPR
@itemx VEC_UNPACK_LO_EXPR
These nodes represent unpacking of the high and low parts of the input vector,
diff --git a/gcc/expr.cc b/gcc/expr.cc
index f8f5cc5a6ca67f291b3c8b7246d593c0be80272f..454d1391b19a7d2aa53f0a88876d1eaf0494de51 100644
--- a/gcc/expr.cc
+++ b/gcc/expr.cc
@@ -9601,8 +9601,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
target, unsignedp);
return target;
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_MULT_EXPR:
/* If first operand is constant, swap them.
Thus the following special case checks need only
@@ -10380,10 +10378,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
return temp;
}
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/gimple-pretty-print.cc b/gcc/gimple-pretty-print.cc
index 300e9d7ed1e7be73f30875e08c461a8880c3134e..d903826894e7f0dfd34dc0caad92eea3caa45e05 100644
--- a/gcc/gimple-pretty-print.cc
+++ b/gcc/gimple-pretty-print.cc
@@ -459,10 +459,6 @@ dump_binary_rhs (pretty_printer *buffer, const gassign *gs, int spc,
case VEC_PACK_FLOAT_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_SERIES_EXPR:
for (p = get_tree_code_name (code); *p; p++)
pp_character (buffer, TOUPPER (*p));
diff --git a/gcc/gimple-range-op.cc b/gcc/gimple-range-op.cc
index 4ca32a7b5d52f8426b09d1446a336650e143b41f..5ae7f7596c6fc6f901e4e47ae44f00185f4602b2 100644
--- a/gcc/gimple-range-op.cc
+++ b/gcc/gimple-range-op.cc
@@ -797,12 +797,6 @@ gimple_range_op_handler::maybe_non_standard ()
if (gimple_code (m_stmt) == GIMPLE_ASSIGN)
switch (gimple_assign_rhs_code (m_stmt))
{
- case WIDEN_PLUS_EXPR:
- {
- signed_op = ptr_op_widen_plus_signed;
- unsigned_op = ptr_op_widen_plus_unsigned;
- }
- gcc_fallthrough ();
case WIDEN_MULT_EXPR:
{
m_valid = false;
diff --git a/gcc/optabs-tree.cc b/gcc/optabs-tree.cc
index 8010046c6a8b3e809c989ddef7a06ddaa68ae32a..ee1aa8c9676ee9c67edbf403e6295da391826a62 100644
--- a/gcc/optabs-tree.cc
+++ b/gcc/optabs-tree.cc
@@ -190,22 +190,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
return (TYPE_UNSIGNED (type)
? vec_widen_ushiftl_lo_optab : vec_widen_sshiftl_lo_optab);
- case VEC_WIDEN_PLUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_lo_optab : vec_widen_saddl_lo_optab);
-
- case VEC_WIDEN_PLUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_hi_optab : vec_widen_saddl_hi_optab);
-
- case VEC_WIDEN_MINUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_lo_optab : vec_widen_ssubl_lo_optab);
-
- case VEC_WIDEN_MINUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_hi_optab : vec_widen_ssubl_hi_optab);
-
case VEC_UNPACK_HI_EXPR:
return (TYPE_UNSIGNED (type)
? vec_unpacku_hi_optab : vec_unpacks_hi_optab);
@@ -312,8 +296,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
'hi'/'lo' pair using codes such as VEC_WIDEN_MINUS_HI/LO.
Supported widening operations:
- WIDEN_MINUS_EXPR
- WIDEN_PLUS_EXPR
WIDEN_MULT_EXPR
WIDEN_LSHIFT_EXPR
@@ -345,12 +327,6 @@ supportable_half_widening_operation (enum tree_code code, tree vectype_out,
case WIDEN_LSHIFT_EXPR:
*code1 = LSHIFT_EXPR;
break;
- case WIDEN_MINUS_EXPR:
- *code1 = MINUS_EXPR;
- break;
- case WIDEN_PLUS_EXPR:
- *code1 = PLUS_EXPR;
- break;
case WIDEN_MULT_EXPR:
*code1 = MULT_EXPR;
break;
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index a9fcc7fd050f871437ef336ecfb8d6cc81280ee0..f80cd1465df83b5540492e619e56b9af249e9f31 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -4017,8 +4017,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case PLUS_EXPR:
case MINUS_EXPR:
{
@@ -4139,10 +4137,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index c702f0032a19203a7c536a01c1e7f47fc7b77add..6e5fd45a0c2435109dd3d50e8fc8e1d4969a1fd0 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4273,8 +4273,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case REALIGN_LOAD_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case DOT_PROD_EXPR:
@@ -4283,10 +4281,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case WIDEN_MULT_MINUS_EXPR:
case WIDEN_LSHIFT_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-pretty-print.cc b/gcc/tree-pretty-print.cc
index 7947f9647a15110b52d195643ad7d28ee32d4236..9941d8bf80535a98e647b8928619a6bf08bc434c 100644
--- a/gcc/tree-pretty-print.cc
+++ b/gcc/tree-pretty-print.cc
@@ -2874,8 +2874,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
break;
/* Binary arithmetic and logic expressions. */
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case MULT_EXPR:
@@ -3831,10 +3829,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
case VEC_SERIES_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
@@ -4352,12 +4346,6 @@ op_symbol_code (enum tree_code code)
case WIDEN_LSHIFT_EXPR:
return "w<<";
- case WIDEN_PLUS_EXPR:
- return "w+";
-
- case WIDEN_MINUS_EXPR:
- return "w-";
-
case POINTER_PLUS_EXPR:
return "+";
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 8daf7bd7dd34d043b1d7b4cba1779f0ecf9f520a..213a3899a6c145bb057cd118bec1df7a05728aef 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -136,8 +136,6 @@ vect_get_smallest_scalar_type (stmt_vec_info stmt_info, tree scalar_type)
|| gimple_assign_rhs_code (assign) == WIDEN_SUM_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_MULT_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_LSHIFT_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_PLUS_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_MINUS_EXPR
|| gimple_assign_rhs_code (assign) == FLOAT_EXPR)
{
tree rhs_type = TREE_TYPE (gimple_assign_rhs1 (assign));
diff --git a/gcc/tree-vect-generic.cc b/gcc/tree-vect-generic.cc
index 445da53292e9d1d2db62ca962fc017bb0e6c9bbe..342ffc5fa7f3b8f37e6bd4658d2f1fccf1d2c7fa 100644
--- a/gcc/tree-vect-generic.cc
+++ b/gcc/tree-vect-generic.cc
@@ -2227,10 +2227,6 @@ expand_vector_operations_1 (gimple_stmt_iterator *gsi,
arguments, not the widened result. VEC_UNPACK_FLOAT_*_EXPR is
calculated in the same way above. */
if (code == WIDEN_SUM_EXPR
- || code == VEC_WIDEN_PLUS_HI_EXPR
- || code == VEC_WIDEN_PLUS_LO_EXPR
- || code == VEC_WIDEN_MINUS_HI_EXPR
- || code == VEC_WIDEN_MINUS_LO_EXPR
|| code == VEC_WIDEN_MULT_HI_EXPR
|| code == VEC_WIDEN_MULT_LO_EXPR
|| code == VEC_WIDEN_MULT_EVEN_EXPR
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 1a514461b2ca416f45a5fa9abe417980d33ef4df..13c69133d7ae565cf0334390cb0c303c89f98ac8 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -561,21 +561,35 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ {
+ rhs_code = gimple_assign_rhs_code (stmt);
+ if (rhs_code.safe_as_tree_code () != code
+ && rhs_code.get_rep () != widened_code.get_rep ())
+ return 0;
+ }
+ else if (is_gimple_call (stmt))
+ {
+ rhs_code = gimple_call_combined_fn (stmt);
+ if (rhs_code.get_rep () != widened_code.get_rep ())
+ return 0;
+ }
+ else
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = gimple_get_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -588,7 +602,7 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op = gimple_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1342,8 +1356,9 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
- false, 2, unprom, &half_type))
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ CFN_VEC_WIDEN_MINUS, false, 2, unprom,
+ &half_type))
return NULL;
vect_pattern_detected ("vect_recog_sad_pattern", last_stmt);
@@ -2696,9 +2711,10 @@ vect_recog_average_pattern (vec_info *vinfo,
internal_fn ifn = IFN_AVG_FLOOR;
vect_unpromoted_value unprom[3];
tree new_type;
+ enum optab_subtype subtype;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
- unprom, &new_type);
+ CFN_VEC_WIDEN_PLUS, false, 3,
+ unprom, &new_type, &subtype);
if (nops == 0)
return NULL;
if (nops == 3)
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index f4806073f48d4dedea3ac9bd855792b152d78919..38f4680d45ab80e8f86327327c13667d96bc5bea 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5032,9 +5032,7 @@ vectorizable_conversion (vec_info *vinfo,
else
return false;
- bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
+ bool widen_arith = (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == IFN_VEC_WIDEN_PLUS
|| code == IFN_VEC_WIDEN_MINUS);
@@ -5086,8 +5084,6 @@ vectorizable_conversion (vec_info *vinfo,
{
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
|| code == IFN_VEC_WIDEN_PLUS
|| code == IFN_VEC_WIDEN_MINUS);
@@ -12211,7 +12207,7 @@ supportable_widening_operation (vec_info *vinfo,
class loop *vect_loop = NULL;
machine_mode vec_mode;
enum insn_code icode1, icode2;
- optab optab1, optab2;
+ optab optab1 = unknown_optab, optab2 = unknown_optab;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
code_helper c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
@@ -12305,16 +12301,6 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_WIDEN_LSHIFT_HI_EXPR;
break;
- case WIDEN_PLUS_EXPR:
- c1 = VEC_WIDEN_PLUS_LO_EXPR;
- c2 = VEC_WIDEN_PLUS_HI_EXPR;
- break;
-
- case WIDEN_MINUS_EXPR:
- c1 = VEC_WIDEN_MINUS_LO_EXPR;
- c2 = VEC_WIDEN_MINUS_HI_EXPR;
- break;
-
CASE_CONVERT:
c1 = VEC_UNPACK_LO_EXPR;
c2 = VEC_UNPACK_HI_EXPR;
diff --git a/gcc/tree.def b/gcc/tree.def
index ee02754354f015a16737c7e879d89c3e3be0d5aa..a58e608a90078818a7ade9d1173ac7ec84c48c7a 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1374,15 +1374,16 @@ DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr", tcc_expression, 3)
DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr", tcc_binary, 2)
/* Widening sad (sum of absolute differences).
- The first two arguments are of type t1 which should be integer.
- The third argument and the result are of type t2, such that t2 is at least
- twice the size of t1. Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
+ The first two arguments are of type t1 which should be a vector of integers.
+ The third argument and the result are of type t2, such that the size of
+ the elements of t2 is at least twice the size of the elements of t1.
+ Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
equivalent to:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = PLUS_EXPR (tmp2, arg3)
or:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = WIDEN_SUM_EXPR (tmp2, arg3)
*/
@@ -1421,8 +1422,6 @@ DEFTREECODE (WIDEN_MULT_MINUS_EXPR, "widen_mult_minus_expr", tcc_expression, 3)
the first argument from type t1 to type t2, and then shifting it
by the second argument. */
DEFTREECODE (WIDEN_LSHIFT_EXPR, "widen_lshift_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_PLUS_EXPR, "widen_plus_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_MINUS_EXPR, "widen_minus_expr", tcc_binary, 2)
/* Widening vector multiplication.
The two operands are vectors with N elements of size S. Multiplying the
@@ -1487,10 +1486,6 @@ DEFTREECODE (VEC_PACK_FLOAT_EXPR, "vec_pack_float_expr", tcc_binary, 2)
*/
DEFTREECODE (VEC_WIDEN_LSHIFT_HI_EXPR, "widen_lshift_hi_expr", tcc_binary, 2)
DEFTREECODE (VEC_WIDEN_LSHIFT_LO_EXPR, "widen_lshift_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_HI_EXPR, "widen_plus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_LO_EXPR, "widen_plus_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_HI_EXPR, "widen_minus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_LO_EXPR, "widen_minus_lo_expr", tcc_binary, 2)
/* PREDICT_EXPR. Specify hint for branch prediction. The
PREDICT_EXPR_PREDICTOR specify predictor and PREDICT_EXPR_OUTCOME the
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2023-04-20 13:23 ` Andre Vieira (lists)
@ 2023-04-24 11:57 ` Richard Biener
2023-04-24 13:01 ` Richard Sandiford
2023-04-25 9:55 ` Andre Vieira (lists)
0 siblings, 2 replies; 53+ messages in thread
From: Richard Biener @ 2023-04-24 11:57 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, Richard Sandiford, gcc-patches
On Thu, Apr 20, 2023 at 3:24 PM Andre Vieira (lists) via Gcc-patches
<gcc-patches@gcc.gnu.org> wrote:
>
> Rebased all three patches and made some small changes to the second one:
> - removed sub and abd optabs from commutative_optab_p, I suspect this
> was a copy paste mistake,
> - removed what I believe to be a superfluous switch case in vectorizable
> conversion, the one that was here:
> + if (code.is_fn_code ())
> + {
> + internal_fn ifn = as_internal_fn (code.as_fn_code ());
> + int ecf_flags = internal_fn_flags (ifn);
> + gcc_assert (ecf_flags & ECF_MULTI);
> +
> + switch (code.as_fn_code ())
> + {
> + case CFN_VEC_WIDEN_PLUS:
> + break;
> + case CFN_VEC_WIDEN_MINUS:
> + break;
> + case CFN_LAST:
> + default:
> + return false;
> + }
> +
> + internal_fn lo, hi;
> + lookup_multi_internal_fn (ifn, &lo, &hi);
> + *code1 = as_combined_fn (lo);
> + *code2 = as_combined_fn (hi);
> + optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
> + optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
> }
>
> I don't think we need to check they are a specfic fn code, as we look-up
> optabs and if they succeed then surely we can vectorize?
>
> OK for trunk?
In the first patch I see some uses of safe_as_tree_code like
+ if (ch.is_tree_code ())
+ return op1 == NULL_TREE ? gimple_build_assign (lhs,
ch.safe_as_tree_code (),
+ op0) :
+ gimple_build_assign (lhs, ch.safe_as_tree_code (),
+ op0, op1);
+ else
+ {
+ internal_fn fn = as_internal_fn (ch.safe_as_fn_code ());
+ gimple* stmt;
where the context actually requires a valid tree code. Please change those
to force to tree code / ifn code. Just use explicit casts here and the other
places that are similar. Before the as_internal_fn just put a
gcc_assert (ch.is_internal_fn ()).
Maybe the need for the (ugly) safe_as_tree_code/fn_code goes away then.
Otherwise patch1 looks OK.
Unfortunately there are no ChangeLog / patch descriptions on the changes.
patch2 has
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ {
+ rhs_code = gimple_assign_rhs_code (stmt);
+ if (rhs_code.safe_as_tree_code () != code
+ && rhs_code.get_rep () != widened_code.get_rep ())
+ return 0;
+ }
+ else if (is_gimple_call (stmt))
+ {
+ rhs_code = gimple_call_combined_fn (stmt);
+ if (rhs_code.get_rep () != widened_code.get_rep ())
+ return 0;
+ }
that looks mightly complicated - esp. the use of get_rep ()
looks dangerous? What's the intent of this? Not that I
understand the existing code much. A comment would
clearly help (also indicating test coverage).
> Kind regards,
> Andre
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2023-04-24 11:57 ` Richard Biener
@ 2023-04-24 13:01 ` Richard Sandiford
2023-04-25 12:30 ` Richard Biener
2023-04-25 9:55 ` Andre Vieira (lists)
1 sibling, 1 reply; 53+ messages in thread
From: Richard Sandiford @ 2023-04-24 13:01 UTC (permalink / raw)
To: Richard Biener; +Cc: Andre Vieira (lists), Richard Biener, gcc-patches
Richard Biener <richard.guenther@gmail.com> writes:
> On Thu, Apr 20, 2023 at 3:24 PM Andre Vieira (lists) via Gcc-patches
> <gcc-patches@gcc.gnu.org> wrote:
>>
>> Rebased all three patches and made some small changes to the second one:
>> - removed sub and abd optabs from commutative_optab_p, I suspect this
>> was a copy paste mistake,
>> - removed what I believe to be a superfluous switch case in vectorizable
>> conversion, the one that was here:
>> + if (code.is_fn_code ())
>> + {
>> + internal_fn ifn = as_internal_fn (code.as_fn_code ());
>> + int ecf_flags = internal_fn_flags (ifn);
>> + gcc_assert (ecf_flags & ECF_MULTI);
>> +
>> + switch (code.as_fn_code ())
>> + {
>> + case CFN_VEC_WIDEN_PLUS:
>> + break;
>> + case CFN_VEC_WIDEN_MINUS:
>> + break;
>> + case CFN_LAST:
>> + default:
>> + return false;
>> + }
>> +
>> + internal_fn lo, hi;
>> + lookup_multi_internal_fn (ifn, &lo, &hi);
>> + *code1 = as_combined_fn (lo);
>> + *code2 = as_combined_fn (hi);
>> + optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
>> + optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
>> }
>>
>> I don't think we need to check they are a specfic fn code, as we look-up
>> optabs and if they succeed then surely we can vectorize?
>>
>> OK for trunk?
>
> In the first patch I see some uses of safe_as_tree_code like
>
> + if (ch.is_tree_code ())
> + return op1 == NULL_TREE ? gimple_build_assign (lhs,
> ch.safe_as_tree_code (),
> + op0) :
> + gimple_build_assign (lhs, ch.safe_as_tree_code (),
> + op0, op1);
> + else
> + {
> + internal_fn fn = as_internal_fn (ch.safe_as_fn_code ());
> + gimple* stmt;
>
> where the context actually requires a valid tree code. Please change those
> to force to tree code / ifn code. Just use explicit casts here and the other
> places that are similar. Before the as_internal_fn just put a
> gcc_assert (ch.is_internal_fn ()).
Also, doesn't the above ?: simplify to the "else" arm? Null trailing
arguments would be ignored for unary operators.
I wasn't sure what to make of the op0 handling:
> +/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
> + or internal_fn contained in ch, respectively. */
> +gimple *
> +vect_gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
> +{
> + if (op0 == NULL_TREE)
> + return NULL;
Can that happen, and if so, does returning null make sense?
Maybe an assert would be safer.
Thanks,
Richard
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2023-04-24 11:57 ` Richard Biener
2023-04-24 13:01 ` Richard Sandiford
@ 2023-04-25 9:55 ` Andre Vieira (lists)
2023-04-28 12:36 ` [PATCH 1/3] Refactor to allow internal_fn's Andre Vieira (lists)
` (2 more replies)
1 sibling, 3 replies; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-04-25 9:55 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
On 24/04/2023 12:57, Richard Biener wrote:
> On Thu, Apr 20, 2023 at 3:24 PM Andre Vieira (lists) via Gcc-patches
> <gcc-patches@gcc.gnu.org> wrote:
>>
>> Rebased all three patches and made some small changes to the second one:
>> - removed sub and abd optabs from commutative_optab_p, I suspect this
>> was a copy paste mistake,
>> - removed what I believe to be a superfluous switch case in vectorizable
>> conversion, the one that was here:
>> + if (code.is_fn_code ())
>> + {
>> + internal_fn ifn = as_internal_fn (code.as_fn_code ());
>> + int ecf_flags = internal_fn_flags (ifn);
>> + gcc_assert (ecf_flags & ECF_MULTI);
>> +
>> + switch (code.as_fn_code ())
>> + {
>> + case CFN_VEC_WIDEN_PLUS:
>> + break;
>> + case CFN_VEC_WIDEN_MINUS:
>> + break;
>> + case CFN_LAST:
>> + default:
>> + return false;
>> + }
>> +
>> + internal_fn lo, hi;
>> + lookup_multi_internal_fn (ifn, &lo, &hi);
>> + *code1 = as_combined_fn (lo);
>> + *code2 = as_combined_fn (hi);
>> + optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
>> + optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
>> }
>>
>> I don't think we need to check they are a specfic fn code, as we look-up
>> optabs and if they succeed then surely we can vectorize?
>>
>> OK for trunk?
>
> In the first patch I see some uses of safe_as_tree_code like
>
> + if (ch.is_tree_code ())
> + return op1 == NULL_TREE ? gimple_build_assign (lhs,
> ch.safe_as_tree_code (),
> + op0) :
> + gimple_build_assign (lhs, ch.safe_as_tree_code (),
> + op0, op1);
> + else
> + {
> + internal_fn fn = as_internal_fn (ch.safe_as_fn_code ());
> + gimple* stmt;
>
> where the context actually requires a valid tree code. Please change those
> to force to tree code / ifn code. Just use explicit casts here and the other
> places that are similar. Before the as_internal_fn just put a
> gcc_assert (ch.is_internal_fn ()).
>
> Maybe the need for the (ugly) safe_as_tree_code/fn_code goes away then.
>
> Otherwise patch1 looks OK.
>
> Unfortunately there are no ChangeLog / patch descriptions on the changes.
> patch2 has
>
> - tree_code rhs_code = gimple_assign_rhs_code (assign);
> - if (rhs_code != code && rhs_code != widened_code)
> + code_helper rhs_code;
> + if (is_gimple_assign (stmt))
> + {
> + rhs_code = gimple_assign_rhs_code (stmt);
> + if (rhs_code.safe_as_tree_code () != code
> + && rhs_code.get_rep () != widened_code.get_rep ())
> + return 0;
> + }
> + else if (is_gimple_call (stmt))
> + {
> + rhs_code = gimple_call_combined_fn (stmt);
> + if (rhs_code.get_rep () != widened_code.get_rep ())
> + return 0;
> + }
>
> that looks mightly complicated - esp. the use of get_rep ()
> looks dangerous? What's the intent of this? Not that I
> understand the existing code much. A comment would
> clearly help (also indicating test coverage).
I don't think the use of get_rep here is dangerous, it's meant to avoid
having to check whether widened_code is the same 'kind' as rhs_code.
With get_rep we don't have to do this check first because tree_codes
will have positive reps and combined_fns negative reps. Having said
that, this can all be simplified and we don't need to use get_rep either
as the == operator has been overloaded to use get_rep and even use the
constructor on the rhs of the ==, so I suggest moving the check after
assigning rhs_code and just doing:
if (rhs_code != code
&& rhs_code != widened_code)
return 0;
>
>
>> Kind regards,
>> Andre
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2023-04-24 13:01 ` Richard Sandiford
@ 2023-04-25 12:30 ` Richard Biener
2023-04-28 16:06 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-04-25 12:30 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Richard Biener, Andre Vieira (lists), gcc-patches
On Mon, 24 Apr 2023, Richard Sandiford wrote:
> Richard Biener <richard.guenther@gmail.com> writes:
> > On Thu, Apr 20, 2023 at 3:24?PM Andre Vieira (lists) via Gcc-patches
> > <gcc-patches@gcc.gnu.org> wrote:
> >>
> >> Rebased all three patches and made some small changes to the second one:
> >> - removed sub and abd optabs from commutative_optab_p, I suspect this
> >> was a copy paste mistake,
> >> - removed what I believe to be a superfluous switch case in vectorizable
> >> conversion, the one that was here:
> >> + if (code.is_fn_code ())
> >> + {
> >> + internal_fn ifn = as_internal_fn (code.as_fn_code ());
> >> + int ecf_flags = internal_fn_flags (ifn);
> >> + gcc_assert (ecf_flags & ECF_MULTI);
> >> +
> >> + switch (code.as_fn_code ())
> >> + {
> >> + case CFN_VEC_WIDEN_PLUS:
> >> + break;
> >> + case CFN_VEC_WIDEN_MINUS:
> >> + break;
> >> + case CFN_LAST:
> >> + default:
> >> + return false;
> >> + }
> >> +
> >> + internal_fn lo, hi;
> >> + lookup_multi_internal_fn (ifn, &lo, &hi);
> >> + *code1 = as_combined_fn (lo);
> >> + *code2 = as_combined_fn (hi);
> >> + optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
> >> + optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
> >> }
> >>
> >> I don't think we need to check they are a specfic fn code, as we look-up
> >> optabs and if they succeed then surely we can vectorize?
> >>
> >> OK for trunk?
> >
> > In the first patch I see some uses of safe_as_tree_code like
> >
> > + if (ch.is_tree_code ())
> > + return op1 == NULL_TREE ? gimple_build_assign (lhs,
> > ch.safe_as_tree_code (),
> > + op0) :
> > + gimple_build_assign (lhs, ch.safe_as_tree_code (),
> > + op0, op1);
> > + else
> > + {
> > + internal_fn fn = as_internal_fn (ch.safe_as_fn_code ());
> > + gimple* stmt;
> >
> > where the context actually requires a valid tree code. Please change those
> > to force to tree code / ifn code. Just use explicit casts here and the other
> > places that are similar. Before the as_internal_fn just put a
> > gcc_assert (ch.is_internal_fn ()).
>
> Also, doesn't the above ?: simplify to the "else" arm? Null trailing
> arguments would be ignored for unary operators.
>
> I wasn't sure what to make of the op0 handling:
>
> > +/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
> > + or internal_fn contained in ch, respectively. */
> > +gimple *
> > +vect_gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
> > +{
> > + if (op0 == NULL_TREE)
> > + return NULL;
>
> Can that happen, and if so, does returning null make sense?
> Maybe an assert would be safer.
Yeah, I was hoping to have a look whether the new gimple_build
overloads could be used to make this all better (but hoped we can
finally get this series in in some way).
Richard.
^ permalink raw reply [flat|nested] 53+ messages in thread
* [PATCH 1/3] Refactor to allow internal_fn's
2023-04-25 9:55 ` Andre Vieira (lists)
@ 2023-04-28 12:36 ` Andre Vieira (lists)
2023-05-03 11:55 ` Richard Biener
2023-04-28 12:37 ` [PATCH 2/3] Refactor widen_plus as internal_fn Andre Vieira (lists)
2023-04-28 12:37 ` [PATCH 3/3] Remove widen_plus/minus_expr tree codes Andre Vieira (lists)
2 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-04-28 12:36 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1769 bytes --]
Hi,
I'm posting the patches separately now with ChangeLogs.
I made the suggested changes and tried to simplify the code a bit
further. Where internal to tree-vect-stmts I changed most functions to
use code_helper to avoid having to check at places we didn't need to. I
was trying to simplify things further by also modifying
supportable_half_widening_operation and supportable_convert_operation
but the result of that was that I ended up moving the code to cast to
tree code inside them rather than at the call site and it didn't look
simpler, so I left those. Though if we did make those changes we'd no
longer need to keep around the tc1 variable in
vectorizable_conversion... Let me know what you think.
gcc/ChangeLog:
2023-04-28 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
* tree-vect-patterns.cc (vect_gimple_build): New Function.
(vect_recog_widen_op_pattern): Refactor to use code_helper.
* tree-vect-stmts.cc (vect_gen_widened_results_half): Likewise.
(vect_create_vectorized_demotion_stmts): Likewise.
(vect_create_vectorized_promotion_stmts): Likewise.
(vect_create_half_widening_stmts): Likewise.
(vectorizable_conversion): Likewise.
(vectorizable_call): Likewise.
(supportable_widening_operation): Likewise.
(supportable_narrowing_operation): Likewise.
(simple_integer_narrowing): Likewise.
* tree-vectorizer.h (supportable_widening_operation): Likewise.
(supportable_narrowing_operation): Likewise.
(vect_gimple_build): New function prototype.
* tree.h (code_helper::safe_as_tree_code): New function.
(code_helper::safe_as_fn_code): New function.
[-- Attachment #2: ifn0_v2.patch --]
[-- Type: text/plain, Size: 22695 bytes --]
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 8802141cd6edb298866025b8a55843eae1f0eb17..b35023adade94c1996cd076c4b7419560e819c6b 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -25,6 +25,8 @@ along with GCC; see the file COPYING3. If not see
#include "rtl.h"
#include "tree.h"
#include "gimple.h"
+#include "gimple-iterator.h"
+#include "gimple-fold.h"
#include "ssa.h"
#include "expmed.h"
#include "optabs-tree.h"
@@ -1391,7 +1393,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
- tree_code orig_code, tree_code wide_code,
+ tree_code orig_code, code_helper wide_code,
bool shift_p, const char *name)
{
gimple *last_stmt = last_stmt_info->stmt;
@@ -1434,7 +1436,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
vecctype = get_vectype_for_scalar_type (vinfo, ctype);
}
- enum tree_code dummy_code;
+ code_helper dummy_code;
int dummy_int;
auto_vec<tree> dummy_vec;
if (!vectype
@@ -1455,8 +1457,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
2, oprnd, half_type, unprom, vectype);
tree var = vect_recog_temp_ssa_var (itype, NULL);
- gimple *pattern_stmt = gimple_build_assign (var, wide_code,
- oprnd[0], oprnd[1]);
+ gimple *pattern_stmt = vect_gimple_build (var, wide_code, oprnd[0], oprnd[1]);
if (vecctype != vecitype)
pattern_stmt = vect_convert_output (vinfo, last_stmt_info, ctype,
@@ -6406,3 +6407,20 @@ vect_pattern_recog (vec_info *vinfo)
/* After this no more add_stmt calls are allowed. */
vinfo->stmt_vec_info_ro = true;
}
+
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple *
+vect_gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
+{
+ gcc_assert (op0 != NULL_TREE);
+ if (ch.is_tree_code ())
+ return gimple_build_assign (lhs, (tree_code) ch, op0, op1);
+
+ gcc_assert (ch.is_internal_fn ());
+ gimple* stmt = gimple_build_call_internal (as_internal_fn ((combined_fn) ch),
+ op1 == NULL_TREE ? 1 : 2,
+ op0, op1);
+ gimple_call_set_lhs (stmt, lhs);
+ return stmt;
+}
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 6b7dbfd4a231baec24e740ffe0ce0b0bf7a1de6b..ce47f4940fa9a1baca4ba1162065cfc3b4072eba 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -3258,13 +3258,13 @@ vectorizable_bswap (vec_info *vinfo,
static bool
simple_integer_narrowing (tree vectype_out, tree vectype_in,
- tree_code *convert_code)
+ code_helper *convert_code)
{
if (!INTEGRAL_TYPE_P (TREE_TYPE (vectype_out))
|| !INTEGRAL_TYPE_P (TREE_TYPE (vectype_in)))
return false;
- tree_code code;
+ code_helper code;
int multi_step_cvt = 0;
auto_vec <tree, 8> interm_types;
if (!supportable_narrowing_operation (NOP_EXPR, vectype_out, vectype_in,
@@ -3478,7 +3478,7 @@ vectorizable_call (vec_info *vinfo,
tree callee = gimple_call_fndecl (stmt);
/* First try using an internal function. */
- tree_code convert_code = ERROR_MARK;
+ code_helper convert_code = MAX_TREE_CODES;
if (cfn != CFN_LAST
&& (modifier == NONE
|| (modifier == NARROW
@@ -3664,8 +3664,8 @@ vectorizable_call (vec_info *vinfo,
continue;
}
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, convert_code,
- prev_res, half_res);
+ new_stmt = vect_gimple_build (new_temp, convert_code,
+ prev_res, half_res);
vect_finish_stmt_generation (vinfo, stmt_info,
new_stmt, gsi);
}
@@ -3755,8 +3755,8 @@ vectorizable_call (vec_info *vinfo,
continue;
}
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, convert_code,
- prev_res, half_res);
+ new_stmt = vect_gimple_build (new_temp, convert_code, prev_res,
+ half_res);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -4768,7 +4768,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
STMT_INFO is the original scalar stmt that we are vectorizing. */
static gimple *
-vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
+vect_gen_widened_results_half (vec_info *vinfo, code_helper ch,
tree vec_oprnd0, tree vec_oprnd1, int op_type,
tree vec_dest, gimple_stmt_iterator *gsi,
stmt_vec_info stmt_info)
@@ -4777,12 +4777,11 @@ vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
tree new_temp;
/* Generate half of the widened result: */
- gcc_assert (op_type == TREE_CODE_LENGTH (code));
if (op_type != binary_op)
vec_oprnd1 = NULL;
- new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0, vec_oprnd1);
+ new_stmt = vect_gimple_build (vec_dest, ch, vec_oprnd0, vec_oprnd1);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
return new_stmt;
@@ -4799,7 +4798,7 @@ vect_create_vectorized_demotion_stmts (vec_info *vinfo, vec<tree> *vec_oprnds,
stmt_vec_info stmt_info,
vec<tree> &vec_dsts,
gimple_stmt_iterator *gsi,
- slp_tree slp_node, enum tree_code code)
+ slp_tree slp_node, code_helper code)
{
unsigned int i;
tree vop0, vop1, new_tmp, vec_dest;
@@ -4811,9 +4810,9 @@ vect_create_vectorized_demotion_stmts (vec_info *vinfo, vec<tree> *vec_oprnds,
/* Create demotion operation. */
vop0 = (*vec_oprnds)[i];
vop1 = (*vec_oprnds)[i + 1];
- gassign *new_stmt = gimple_build_assign (vec_dest, code, vop0, vop1);
+ gimple *new_stmt = vect_gimple_build (vec_dest, code, vop0, vop1);
new_tmp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_tmp);
+ gimple_set_lhs (new_stmt, new_tmp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
if (multi_step_cvt)
@@ -4861,8 +4860,8 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
- enum tree_code code2, int op_type)
+ code_helper ch1,
+ code_helper ch2, int op_type)
{
int i;
tree vop0, vop1, new_tmp1, new_tmp2;
@@ -4878,10 +4877,10 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vop1 = NULL_TREE;
/* Generate the two halves of promotion operation. */
- new_stmt1 = vect_gen_widened_results_half (vinfo, code1, vop0, vop1,
+ new_stmt1 = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
- new_stmt2 = vect_gen_widened_results_half (vinfo, code2, vop0, vop1,
+ new_stmt2 = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
if (is_gimple_call (new_stmt1))
@@ -4912,7 +4911,7 @@ vect_create_half_widening_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
+ code_helper code1,
int op_type)
{
int i;
@@ -4942,13 +4941,13 @@ vect_create_half_widening_stmts (vec_info *vinfo,
new_stmt2 = gimple_build_assign (new_tmp2, NOP_EXPR, vop1);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt2, gsi);
/* Perform the operation. With both vector inputs widened. */
- new_stmt3 = gimple_build_assign (vec_dest, code1, new_tmp1, new_tmp2);
+ new_stmt3 = vect_gimple_build (vec_dest, code1, new_tmp1, new_tmp2);
}
else
{
/* Perform the operation. With the single vector input widened. */
- new_stmt3 = gimple_build_assign (vec_dest, code1, new_tmp1, vop1);
- }
+ new_stmt3 = vect_gimple_build (vec_dest, code1, new_tmp1, vop1);
+ }
new_tmp3 = make_ssa_name (vec_dest, new_stmt3);
gimple_assign_set_lhs (new_stmt3, new_tmp3);
@@ -4978,8 +4977,9 @@ vectorizable_conversion (vec_info *vinfo,
tree scalar_dest;
tree op0, op1 = NULL_TREE;
loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
- enum tree_code code, code1 = ERROR_MARK, code2 = ERROR_MARK;
- enum tree_code codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
+ tree_code tc1;
+ code_helper code, code1, code2;
+ code_helper codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
tree new_temp;
enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type};
int ndts = 2;
@@ -5008,31 +5008,43 @@ vectorizable_conversion (vec_info *vinfo,
&& ! vec_stmt)
return false;
- gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
- if (!stmt)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return false;
- if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
+ if (gimple_get_lhs (stmt) == NULL_TREE
+ || TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- code = gimple_assign_rhs_code (stmt);
- if (!CONVERT_EXPR_CODE_P (code)
- && code != FIX_TRUNC_EXPR
- && code != FLOAT_EXPR
- && code != WIDEN_PLUS_EXPR
- && code != WIDEN_MINUS_EXPR
- && code != WIDEN_MULT_EXPR
- && code != WIDEN_LSHIFT_EXPR)
+ if (TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
+ return false;
+
+ if (is_gimple_assign (stmt))
+ {
+ code = gimple_assign_rhs_code (stmt);
+ op_type = TREE_CODE_LENGTH ((tree_code) code);
+ }
+ else if (gimple_call_internal_p (stmt))
+ {
+ code = gimple_call_internal_fn (stmt);
+ op_type = gimple_call_num_args (stmt);
+ }
+ else
return false;
bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
- op_type = TREE_CODE_LENGTH (code);
+ || code == WIDEN_MINUS_EXPR
+ || code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR);
+
+ if (!widen_arith
+ && !CONVERT_EXPR_CODE_P (code)
+ && code != FIX_TRUNC_EXPR
+ && code != FLOAT_EXPR)
+ return false;
/* Check types of lhs and rhs. */
- scalar_dest = gimple_assign_lhs (stmt);
+ scalar_dest = gimple_get_lhs (stmt);
lhs_type = TREE_TYPE (scalar_dest);
vectype_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -5070,10 +5082,14 @@ vectorizable_conversion (vec_info *vinfo,
if (op_type == binary_op)
{
- gcc_assert (code == WIDEN_MULT_EXPR || code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR || code == WIDEN_MINUS_EXPR);
+ gcc_assert (code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR
+ || code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR);
+
- op1 = gimple_assign_rhs2 (stmt);
+ op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
+ gimple_call_arg (stmt, 0);
tree vectype1_in;
if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
&op1, &slp_op1, &dt[1], &vectype1_in))
@@ -5157,8 +5173,13 @@ vectorizable_conversion (vec_info *vinfo,
&& code != FLOAT_EXPR
&& !CONVERT_EXPR_CODE_P (code))
return false;
- if (supportable_convert_operation (code, vectype_out, vectype_in, &code1))
+ gcc_assert (code.is_tree_code ());
+ if (supportable_convert_operation ((tree_code) code, vectype_out,
+ vectype_in, &tc1))
+ {
+ code1 = tc1;
break;
+ }
/* FALLTHRU */
unsupported:
if (dump_enabled_p ())
@@ -5169,9 +5190,12 @@ vectorizable_conversion (vec_info *vinfo,
case WIDEN:
if (known_eq (nunits_in, nunits_out))
{
- if (!supportable_half_widening_operation (code, vectype_out,
- vectype_in, &code1))
+ if (!(code.is_tree_code ()
+ && supportable_half_widening_operation ((tree_code) code,
+ vectype_out, vectype_in,
+ &tc1)))
goto unsupported;
+ code1 = tc1;
gcc_assert (!(multi_step_cvt && op_type == binary_op));
break;
}
@@ -5205,14 +5229,17 @@ vectorizable_conversion (vec_info *vinfo,
if (GET_MODE_SIZE (rhs_mode) == fltsz)
{
- if (!supportable_convert_operation (code, vectype_out,
- cvt_type, &codecvt1))
+ tc1 = ERROR_MARK;
+ gcc_assert (code.is_tree_code ());
+ if (!supportable_convert_operation ((tree_code) code, vectype_out,
+ cvt_type, &tc1))
goto unsupported;
+ codecvt1 = tc1;
}
- else if (!supportable_widening_operation (vinfo, code, stmt_info,
- vectype_out, cvt_type,
- &codecvt1, &codecvt2,
- &multi_step_cvt,
+ else if (!supportable_widening_operation (vinfo, code,
+ stmt_info, vectype_out,
+ cvt_type, &codecvt1,
+ &codecvt2, &multi_step_cvt,
&interm_types))
continue;
else
@@ -5220,8 +5247,9 @@ vectorizable_conversion (vec_info *vinfo,
if (supportable_widening_operation (vinfo, NOP_EXPR, stmt_info,
cvt_type,
- vectype_in, &code1, &code2,
- &multi_step_cvt, &interm_types))
+ vectype_in, &code1,
+ &code2, &multi_step_cvt,
+ &interm_types))
{
found_mode = true;
break;
@@ -5257,9 +5285,11 @@ vectorizable_conversion (vec_info *vinfo,
cvt_type = get_same_sized_vectype (cvt_type, vectype_in);
if (cvt_type == NULL_TREE)
goto unsupported;
- if (!supportable_convert_operation (code, cvt_type, vectype_in,
- &codecvt1))
+ if (!code.is_tree_code ()
+ || !supportable_convert_operation ((tree_code) code, cvt_type,
+ vectype_in, &tc1))
goto unsupported;
+ codecvt1 = tc1;
if (supportable_narrowing_operation (NOP_EXPR, vectype_out, cvt_type,
&code1, &multi_step_cvt,
&interm_types))
@@ -5377,10 +5407,9 @@ vectorizable_conversion (vec_info *vinfo,
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
/* Arguments are ready, create the new vector stmt. */
- gcc_assert (TREE_CODE_LENGTH (code1) == unary_op);
- gassign *new_stmt = gimple_build_assign (vec_dest, code1, vop0);
+ gimple *new_stmt = vect_gimple_build (vec_dest, code1, vop0);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
if (slp_node)
@@ -5410,17 +5439,16 @@ vectorizable_conversion (vec_info *vinfo,
for (i = multi_step_cvt; i >= 0; i--)
{
tree this_dest = vec_dsts[i];
- enum tree_code c1 = code1, c2 = code2;
+ code_helper c1 = code1, c2 = code2;
if (i == 0 && codecvt2 != ERROR_MARK)
{
c1 = codecvt1;
c2 = codecvt2;
}
if (known_eq (nunits_out, nunits_in))
- vect_create_half_widening_stmts (vinfo, &vec_oprnds0,
- &vec_oprnds1, stmt_info,
- this_dest, gsi,
- c1, op_type);
+ vect_create_half_widening_stmts (vinfo, &vec_oprnds0, &vec_oprnds1,
+ stmt_info, this_dest, gsi, c1,
+ op_type);
else
vect_create_vectorized_promotion_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
@@ -5433,9 +5461,8 @@ vectorizable_conversion (vec_info *vinfo,
gimple *new_stmt;
if (cvt_type)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, codecvt1, vop0);
+ new_stmt = vect_gimple_build (new_temp, codecvt1, vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -5459,10 +5486,8 @@ vectorizable_conversion (vec_info *vinfo,
if (cvt_type)
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- gassign *new_stmt
- = gimple_build_assign (new_temp, codecvt1, vop0);
+ gimple *new_stmt = vect_gimple_build (new_temp, codecvt1, vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
vec_oprnds0[i] = new_temp;
}
@@ -12151,9 +12176,11 @@ vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype)
bool
supportable_widening_operation (vec_info *vinfo,
- enum tree_code code, stmt_vec_info stmt_info,
+ code_helper code,
+ stmt_vec_info stmt_info,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, enum tree_code *code2,
+ code_helper *code1,
+ code_helper *code2,
int *multi_step_cvt,
vec<tree> *interm_types)
{
@@ -12164,7 +12191,7 @@ supportable_widening_operation (vec_info *vinfo,
optab optab1, optab2;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
- enum tree_code c1, c2;
+ tree_code c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
int i;
tree prev_type, intermediate_type;
machine_mode intermediate_mode, prev_mode;
@@ -12174,8 +12201,12 @@ supportable_widening_operation (vec_info *vinfo,
if (loop_info)
vect_loop = LOOP_VINFO_LOOP (loop_info);
- switch (code)
+ switch (code.safe_as_tree_code ())
{
+ case MAX_TREE_CODES:
+ /* Don't set c1 and c2 if code is not a tree_code. */
+ break;
+
case WIDEN_MULT_EXPR:
/* The result of a vectorized widening operation usually requires
two vectors (because the widened results do not fit into one vector).
@@ -12215,8 +12246,9 @@ supportable_widening_operation (vec_info *vinfo,
&& !nested_in_vect_loop_p (vect_loop, stmt_info)
&& supportable_widening_operation (vinfo, VEC_WIDEN_MULT_EVEN_EXPR,
stmt_info, vectype_out,
- vectype_in, code1, code2,
- multi_step_cvt, interm_types))
+ vectype_in, code1,
+ code2, multi_step_cvt,
+ interm_types))
{
/* Elements in a vector with vect_used_by_reduction property cannot
be reordered if the use chain with this property does not have the
@@ -12292,7 +12324,7 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
}
- else if (CONVERT_EXPR_CODE_P (code)
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
&& VECTOR_BOOLEAN_TYPE_P (wide_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
&& TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
@@ -12317,8 +12349,12 @@ supportable_widening_operation (vec_info *vinfo,
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- *code1 = c1;
- *code2 = c2;
+ if (code.is_tree_code ())
+ {
+ *code1 = c1;
+ *code2 = c2;
+ }
+
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
@@ -12339,7 +12375,7 @@ supportable_widening_operation (vec_info *vinfo,
prev_type = vectype;
prev_mode = vec_mode;
- if (!CONVERT_EXPR_CODE_P (code))
+ if (!CONVERT_EXPR_CODE_P ((tree_code) code))
return false;
/* We assume here that there will not be more than MAX_INTERM_CVT_STEPS
@@ -12437,9 +12473,9 @@ supportable_widening_operation (vec_info *vinfo,
narrowing operation (short in the above example). */
bool
-supportable_narrowing_operation (enum tree_code code,
+supportable_narrowing_operation (code_helper code,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, int *multi_step_cvt,
+ code_helper *code1, int *multi_step_cvt,
vec<tree> *interm_types)
{
machine_mode vec_mode;
@@ -12454,8 +12490,11 @@ supportable_narrowing_operation (enum tree_code code,
unsigned HOST_WIDE_INT n_elts;
bool uns;
+ if (!code.is_tree_code ())
+ return false;
+
*multi_step_cvt = 0;
- switch (code)
+ switch ((tree_code) code)
{
CASE_CONVERT:
c1 = VEC_PACK_TRUNC_EXPR;
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 9cf2fb23fe397b467d89aa7cc5ebeaa293ed4cce..f215cd0639bcf803c9d0554cfdc57823431991d5 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2139,13 +2139,12 @@ extern bool vect_is_simple_use (vec_info *, stmt_vec_info, slp_tree,
enum vect_def_type *,
tree *, stmt_vec_info * = NULL);
extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
-extern bool supportable_widening_operation (vec_info *,
- enum tree_code, stmt_vec_info,
- tree, tree, enum tree_code *,
- enum tree_code *, int *,
- vec<tree> *);
-extern bool supportable_narrowing_operation (enum tree_code, tree, tree,
- enum tree_code *, int *,
+extern bool supportable_widening_operation (vec_info*, code_helper,
+ stmt_vec_info, tree, tree,
+ code_helper*, code_helper*,
+ int*, vec<tree> *);
+extern bool supportable_narrowing_operation (code_helper, tree, tree,
+ code_helper *, int *,
vec<tree> *);
extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
@@ -2583,4 +2582,7 @@ vect_is_integer_truncation (stmt_vec_info stmt_info)
&& TYPE_PRECISION (lhs_type) < TYPE_PRECISION (rhs_type));
}
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple * vect_gimple_build (tree, code_helper, tree, tree = NULL_TREE);
#endif /* GCC_TREE_VECTORIZER_H */
diff --git a/gcc/tree.h b/gcc/tree.h
index abcdb5638d49aea4ccc46efa8e540b1fa78aa27a..f6cd528e7d789c3f81fb2da3c1e1a29fa11f6e0f 100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -93,6 +93,8 @@ public:
bool is_internal_fn () const;
bool is_builtin_fn () const;
int get_rep () const { return rep; }
+ enum tree_code safe_as_tree_code () const;
+ combined_fn safe_as_fn_code () const;
bool operator== (const code_helper &other) { return rep == other.rep; }
bool operator!= (const code_helper &other) { return rep != other.rep; }
bool operator== (tree_code c) { return rep == code_helper (c).rep; }
@@ -102,6 +104,17 @@ private:
int rep;
};
+inline enum tree_code
+code_helper::safe_as_tree_code () const
+{
+ return is_tree_code () ? (tree_code) *this : MAX_TREE_CODES;
+}
+
+inline combined_fn
+code_helper::safe_as_fn_code () const {
+ return is_fn_code () ? (combined_fn) *this : CFN_LAST;
+}
+
inline code_helper::operator internal_fn () const
{
return as_internal_fn (combined_fn (*this));
^ permalink raw reply [flat|nested] 53+ messages in thread
* [PATCH 2/3] Refactor widen_plus as internal_fn
2023-04-25 9:55 ` Andre Vieira (lists)
2023-04-28 12:36 ` [PATCH 1/3] Refactor to allow internal_fn's Andre Vieira (lists)
@ 2023-04-28 12:37 ` Andre Vieira (lists)
2023-05-03 12:11 ` Richard Biener
2023-04-28 12:37 ` [PATCH 3/3] Remove widen_plus/minus_expr tree codes Andre Vieira (lists)
2 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-04-28 12:37 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 3598 bytes --]
This patch replaces the existing tree_code widen_plus and widen_minus
patterns with internal_fn versions.
DEF_INTERNAL_OPTAB_HILO_FN is like DEF_INTERNAL_OPTAB_FN except it
provides convenience wrappers for defining conversions that require a
hi/lo split, like widening and narrowing operations. Each definition
for <NAME> will require an optab named <OPTAB> and two other optabs that
you specify for signed and unsigned. The hi/lo pair is necessary because
the widening operations take n narrow elements as inputs and return n/2
wide elements as outputs. The 'lo' operation operates on the first n/2
elements of input. The 'hi' operation operates on the second n/2
elements of input. Defining an internal_fn along with hi/lo variations
allows a single internal function to be returned from a vect_recog
function that will later be expanded to hi/lo.
DEF_INTERNAL_OPTAB_HILO_FN is used in internal-fn.def to register a
widening internal_fn. It is defined differently in different places and
internal-fn.def is sourced from those places so the parameters given can
be reused.
internal-fn.c: defined to expand to hi/lo signed/unsigned optabs,
later defined to generate the 'expand_' functions for the hi/lo
versions of the fn.
internal-fn.def: defined to invoke DEF_INTERNAL_OPTAB_FN for the
original and hi/lo variants of the internal_fn
For example:
IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>addl_hi_<mode> ->
(u/s)addl2
IFN_VEC_WIDEN_PLUS_LO ->
vec_widen_<su>addl_lo_<mode> -> (u/s)addl
This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS
tree codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
gcc/ChangeLog:
2023-04-28 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
Tamar Christina <tamar.christina@arm.com>
* internal-fn.cc (INCLUDE_MAP): Include maps for use in optab
lookup.
(DEF_INTERNAL_OPTAB_HILO_FN): Macro to define an internal_fn that
expands into multiple internal_fns (for widening).
(ifn_cmp): Function to compare ifn's for sorting/searching.
(lookup_hilo_ifn_optab): Add lookup function.
(lookup_hilo_internal_fn): Add lookup function.
(commutative_binary_fn_p): Add widen_plus fn's.
(widening_fn_p): New function.
(decomposes_to_hilo_fn_p): New function.
* internal-fn.def (DEF_INTERNAL_OPTAB_HILO_FN): Define widening
plus,minus functions.
(VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS tree code.
(VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS tree code.
* internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
(lookup_hilo_ifn_optab): Add prototype.
(lookup_hilo_internal_fn): Likewise.
(widening_fn_p): Likewise.
(decomposes_to_hilo_fn_p): Likewise.
* optabs.cc (commutative_optab_p): Add widening plus, minus optabs.
* optabs.def (OPTAB_CD): widen add, sub optabs
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
patterns with a hi/lo split.
(vect_recog_widen_plus_pattern): Refactor to return
IFN_VECT_WIDEN_PLUS.
(vect_recog_widen_minus_pattern): Refactor to return new
IFN_VEC_WIDEN_MINUS.
* tree-vect-stmts.cc (vectorizable_conversion): Add widen plus/minus
ifn
support.
(supportable_widening_operation): Add widen plus/minus ifn support.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-widen-add.c: Test that new
IFN_VEC_WIDEN_PLUS is being used.
* gcc.target/aarch64/vect-widen-sub.c: Test that new
IFN_VEC_WIDEN_MINUS is being used.
[-- Attachment #2: ifn1_v2.patch --]
[-- Type: text/plain, Size: 18412 bytes --]
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 6e81dc05e0e0714256759b0594816df451415a2d..e4d815cd577d266d2bccf6fb68d62aac91a8b4cf 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -17,6 +17,7 @@ You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3. If not see
<http://www.gnu.org/licenses/>. */
+#define INCLUDE_MAP
#include "config.h"
#include "system.h"
#include "coretypes.h"
@@ -70,6 +71,26 @@ const int internal_fn_flags_array[] = {
0
};
+const enum internal_fn internal_fn_hilo_keys_array[] = {
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ IFN_##NAME##_LO, \
+ IFN_##NAME##_HI,
+#include "internal-fn.def"
+ IFN_LAST
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+};
+
+const optab internal_fn_hilo_values_array[] = {
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ SOPTAB##_lo_optab, UOPTAB##_lo_optab, \
+ SOPTAB##_hi_optab, UOPTAB##_hi_optab,
+#include "internal-fn.def"
+ unknown_optab, unknown_optab
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+};
+
/* Return the internal function called NAME, or IFN_LAST if there's
no such function. */
@@ -90,6 +111,61 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+static int
+ifn_cmp (const void *a_, const void *b_)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ auto *a = (const std::pair<ifn_pair, optab> *)a_;
+ auto *b = (const std::pair<ifn_pair, optab> *)b_;
+ return (int) (a->first.first) - (b->first.first);
+}
+
+/* Return the optab belonging to the given internal function NAME for the given
+ SIGN or unknown_optab. */
+
+optab
+lookup_hilo_ifn_optab (enum internal_fn fn, unsigned sign)
+{
+ typedef std::pair<enum internal_fn, unsigned> ifn_pair;
+ typedef auto_vec <std::pair<ifn_pair, optab>>fn_to_optab_map_type;
+ static fn_to_optab_map_type *fn_to_optab_map;
+
+ if (!fn_to_optab_map)
+ {
+ unsigned num
+ = sizeof (internal_fn_hilo_keys_array) / sizeof (enum internal_fn);
+ fn_to_optab_map = new fn_to_optab_map_type ();
+ for (unsigned int i = 0; i < num - 1; ++i)
+ {
+ enum internal_fn fn = internal_fn_hilo_keys_array[i];
+ optab v1 = internal_fn_hilo_values_array[2*i];
+ optab v2 = internal_fn_hilo_values_array[2*i + 1];
+ ifn_pair key1 (fn, 0);
+ fn_to_optab_map->safe_push ({key1, v1});
+ ifn_pair key2 (fn, 1);
+ fn_to_optab_map->safe_push ({key2, v2});
+ }
+ fn_to_optab_map->qsort (ifn_cmp);
+ }
+
+ ifn_pair new_pair (fn, sign ? 1 : 0);
+ optab tmp;
+ std::pair<ifn_pair,optab> pair_wrap (new_pair, tmp);
+ auto entry = fn_to_optab_map->bsearch (&pair_wrap, ifn_cmp);
+ return entry != fn_to_optab_map->end () ? entry->second : unknown_optab;
+}
+
+extern void
+lookup_hilo_internal_fn (enum internal_fn ifn, enum internal_fn *lo,
+ enum internal_fn *hi)
+{
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -3970,6 +4046,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -4043,6 +4122,42 @@ first_commutative_argument (internal_fn fn)
}
}
+/* Return true if FN has a wider output type than its argument types. */
+
+bool
+widening_fn_p (internal_fn fn)
+{
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_MINUS:
+ return true;
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if FN decomposes to _hi and _lo IFN. If true this should also
+ be a widening function. */
+
+bool
+decomposes_to_hilo_fn_p (internal_fn fn)
+{
+ if (!widening_fn_p (fn))
+ return false;
+
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_MINUS:
+ return true;
+
+ default:
+ return false;
+ }
+}
+
/* Return true if IFN_SET_EDOM is supported. */
bool
@@ -4055,6 +4170,32 @@ set_edom_supported_p (void)
#endif
}
+#undef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(CODE, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ static void \
+ expand_##CODE (internal_fn, gcall *) \
+ { \
+ gcc_unreachable (); \
+ } \
+ static void \
+ expand_##CODE##_LO (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_lo##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_lo##_optab); \
+ } \
+ static void \
+ expand_##CODE##_HI (internal_fn fn, gcall *stmt) \
+ { \
+ tree ty = TREE_TYPE (gimple_get_lhs (stmt)); \
+ if (!TYPE_UNSIGNED (ty)) \
+ expand_##TYPE##_optab_fn (fn, stmt, SOPTAB##_hi##_optab); \
+ else \
+ expand_##TYPE##_optab_fn (fn, stmt, UOPTAB##_hi##_optab); \
+ }
+
#define DEF_INTERNAL_OPTAB_FN(CODE, FLAGS, OPTAB, TYPE) \
static void \
expand_##CODE (internal_fn fn, gcall *stmt) \
@@ -4071,6 +4212,7 @@ set_edom_supported_p (void)
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_HILO_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 7fe742c2ae713e7152ab05cfdfba86e4e0aa3456..347ed667d92620e0ee3ea15c58ecac6c242ebe73 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -85,6 +85,13 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_OPTAB_HILO_FN is like DEF_INTERNAL_OPTAB_FN except it
+ provides convenience wrappers for defining conversions that require a
+ hi/lo split, like widening and narrowing operations. Each definition
+ for <NAME> will require an optab named <OPTAB> and two other optabs that
+ you specify for signed and unsigned.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -123,6 +130,14 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_OPTAB_HILO_FN
+#define DEF_INTERNAL_OPTAB_HILO_FN(NAME, FLAGS, OPTAB, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, unknown, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, unknown, TYPE)
+#endif
+
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -315,6 +330,14 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_OPTAB_HILO_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_NOTHROW,
+ vec_widen_add, vec_widen_saddl, vec_widen_uaddl,
+ binary)
+DEF_INTERNAL_OPTAB_HILO_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_NOTHROW,
+ vec_widen_sub, vec_widen_ssubl, vec_widen_usubl,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 08922ed4254898f5fffca3f33973e96ed9ce772f..6a5f8762e872ad2ef64ce2986a678e3b40622d81 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,9 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern optab lookup_hilo_ifn_optab (enum internal_fn, unsigned);
+extern void lookup_hilo_internal_fn (enum internal_fn, enum internal_fn *,
+ enum internal_fn *);
/* Return the ECF_* flags for function FN. */
@@ -210,6 +217,8 @@ extern bool commutative_binary_fn_p (internal_fn);
extern bool commutative_ternary_fn_p (internal_fn);
extern int first_commutative_argument (internal_fn);
extern bool associative_binary_fn_p (internal_fn);
+extern bool widening_fn_p (internal_fn);
+extern bool decomposes_to_hilo_fn_p (internal_fn);
extern bool set_edom_supported_p (void);
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index c8e39c82d57a7d726e7da33d247b80f32ec9236c..d4dd7ee3d34d01c32ab432ae4e4ce9e4b522b2f7 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,12 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_add_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 695f5911b300c9ca5737de9be809fa01aabe5e01..e064189103b3be70644468d11f3c91ac45ffe0d0 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -78,6 +78,8 @@ OPTAB_CD(smsub_widen_optab, "msub$b$a4")
OPTAB_CD(umsub_widen_optab, "umsub$b$a4")
OPTAB_CD(ssmsub_widen_optab, "ssmsub$b$a4")
OPTAB_CD(usmsub_widen_optab, "usmsub$a$b4")
+OPTAB_CD(vec_widen_add_optab, "add$a$b3")
+OPTAB_CD(vec_widen_sub_optab, "sub$a$b3")
OPTAB_CD(vec_load_lanes_optab, "vec_load_lanes$a$b")
OPTAB_CD(vec_store_lanes_optab, "vec_store_lanes$a$b")
OPTAB_CD(vec_mask_load_lanes_optab, "vec_mask_load_lanes$a$b")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index b35023adade94c1996cd076c4b7419560e819c6b..3175dd92187c0935f78ebbf2eb476bdcf8b4ccd1 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1394,14 +1394,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1467,6 +1469,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ enum optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1480,26 +1496,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ enum optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_popcount_pattern
@@ -6067,6 +6087,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index ce47f4940fa9a1baca4ba1162065cfc3b4072eba..2a7ef2439e12d1966e8884433963a3d387a856b7 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5035,7 +5035,9 @@ vectorizable_conversion (vec_info *vinfo,
bool widen_arith = (code == WIDEN_PLUS_EXPR
|| code == WIDEN_MINUS_EXPR
|| code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
+ || code == WIDEN_LSHIFT_EXPR
+ || code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -5085,7 +5087,9 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
+ || code == WIDEN_MINUS_EXPR
+ || code == IFN_VEC_WIDEN_PLUS
+ || code == IFN_VEC_WIDEN_MINUS);
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
@@ -12335,12 +12339,46 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
- {
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn ((combined_fn) code);
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ internal_fn lo, hi;
+ lookup_hilo_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = lookup_hilo_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_hilo_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
}
+ if (code.is_tree_code ())
+ {
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1, vectype, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ }
+ }
+
if (!optab1 || !optab2)
return false;
^ permalink raw reply [flat|nested] 53+ messages in thread
* [PATCH 3/3] Remove widen_plus/minus_expr tree codes
2023-04-25 9:55 ` Andre Vieira (lists)
2023-04-28 12:36 ` [PATCH 1/3] Refactor to allow internal_fn's Andre Vieira (lists)
2023-04-28 12:37 ` [PATCH 2/3] Refactor widen_plus as internal_fn Andre Vieira (lists)
@ 2023-04-28 12:37 ` Andre Vieira (lists)
2023-05-03 12:29 ` Richard Biener
2 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-04-28 12:37 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]
This is a rebase of Joel's previous patch.
This patch removes the old widen plus/minus tree codes which have been
replaced by internal functions.
gcc/ChangeLog:
2023-04-28 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
* doc/generic.texi: Remove old tree codes.
* expr.cc (expand_expr_real_2): Remove old tree code cases.
* gimple-pretty-print.cc (dump_binary_rhs): Likewise.
* optabs-tree.cc (optab_for_tree_code): Likewise.
(supportable_half_widening_operation): Likewise.
* tree-cfg.cc (verify_gimple_assign_binary): Likewise.
* tree-inline.cc (estimate_operator_cost): Likewise.
(op_symbol_code): Likewise.
* tree-vect-data-refs.cc (vect_get_smallest_scalar_type): Likewise.
(vect_analyze_data_ref_accesses): Likewise.
* tree-vect-generic.cc (expand_vector_operations_1): Likewise.
* cfgexpand.cc (expand_debug_expr): Likewise.
* tree-vect-stmts.cc (vectorizable_conversion): Likewise.
(supportable_widening_operation): Likewise.
* gimple-range-op.cc (gimple_range_op_handler::maybe_non_standard):
Likewise.
* tree-vect-patterns.cc (vect_widened_op_tree): Refactor to replace
usage in vect_recog_sad_pattern.
(vect_recog_sad_pattern): Replace tree code widening pattern with
internal function.
(vect_recog_average_pattern): Likewise.
* tree-pretty-print.cc (dump_generic_node): Remove tree code definition.
* tree.def (WIDEN_PLUS_EXPR, WIDEN_MINUS_EXPR, VEC_WIDEN_PLUS_HI_EXPR,
VEC_WIDEN_PLUS_LO_EXPR, VEC_WIDEN_MINUS_HI_EXPR,
VEC_WIDEN_MINUS_LO_EXPR): Likewise
[-- Attachment #2: ifn2_v2.patch --]
[-- Type: text/plain, Size: 19090 bytes --]
diff --git a/gcc/cfgexpand.cc b/gcc/cfgexpand.cc
index 1a1b26b1c6c23ce273bcd08dc9a973f777174007..25b1558dcb941ea491a19aeeb2cd8f4d2dbdf7c6 100644
--- a/gcc/cfgexpand.cc
+++ b/gcc/cfgexpand.cc
@@ -5365,10 +5365,6 @@ expand_debug_expr (tree exp)
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_PERM_EXPR:
case VEC_DUPLICATE_EXPR:
case VEC_SERIES_EXPR:
@@ -5405,8 +5401,6 @@ expand_debug_expr (tree exp)
case WIDEN_MULT_EXPR:
case WIDEN_MULT_PLUS_EXPR:
case WIDEN_MULT_MINUS_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
if (SCALAR_INT_MODE_P (GET_MODE (op0))
&& SCALAR_INT_MODE_P (mode))
{
@@ -5419,10 +5413,6 @@ expand_debug_expr (tree exp)
op1 = simplify_gen_unary (ZERO_EXTEND, mode, op1, inner_mode);
else
op1 = simplify_gen_unary (SIGN_EXTEND, mode, op1, inner_mode);
- if (TREE_CODE (exp) == WIDEN_PLUS_EXPR)
- return simplify_gen_binary (PLUS, mode, op0, op1);
- else if (TREE_CODE (exp) == WIDEN_MINUS_EXPR)
- return simplify_gen_binary (MINUS, mode, op0, op1);
op0 = simplify_gen_binary (MULT, mode, op0, op1);
if (TREE_CODE (exp) == WIDEN_MULT_EXPR)
return op0;
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index 2c14b7abce2db0a3da0a21e916907947cb56a265..3816abaaf4d364d604a44942317f96f3f303e5b6 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,10 +1811,6 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1861,33 +1857,6 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
-@item VEC_WIDEN_PLUS_HI_EXPR
-@itemx VEC_WIDEN_PLUS_LO_EXPR
-These nodes represent widening vector addition of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The result
-is a vector that contains half as many elements, of an integral type whose size
-is twice as wide. In the case of @code{VEC_WIDEN_PLUS_HI_EXPR} the high
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products. In the case of @code{VEC_WIDEN_PLUS_LO_EXPR} the low
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products.
-
-@item VEC_WIDEN_MINUS_HI_EXPR
-@itemx VEC_WIDEN_MINUS_LO_EXPR
-These nodes represent widening vector subtraction of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The high/low
-elements of the second vector are subtracted from the high/low elements of the
-first. The result is a vector that contains half as many elements, of an
-integral type whose size is twice as wide. In the case of
-@code{VEC_WIDEN_MINUS_HI_EXPR} the high @code{N/2} elements of the second
-vector are subtracted from the high @code{N/2} of the first to produce the
-vector of @code{N/2} products. In the case of
-@code{VEC_WIDEN_MINUS_LO_EXPR} the low @code{N/2} elements of the second
-vector are subtracted from the low @code{N/2} of the first to produce the
-vector of @code{N/2} products.
-
@item VEC_UNPACK_HI_EXPR
@itemx VEC_UNPACK_LO_EXPR
These nodes represent unpacking of the high and low parts of the input vector,
diff --git a/gcc/expr.cc b/gcc/expr.cc
index f8f5cc5a6ca67f291b3c8b7246d593c0be80272f..454d1391b19a7d2aa53f0a88876d1eaf0494de51 100644
--- a/gcc/expr.cc
+++ b/gcc/expr.cc
@@ -9601,8 +9601,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
target, unsignedp);
return target;
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_MULT_EXPR:
/* If first operand is constant, swap them.
Thus the following special case checks need only
@@ -10380,10 +10378,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
return temp;
}
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/gimple-pretty-print.cc b/gcc/gimple-pretty-print.cc
index 300e9d7ed1e7be73f30875e08c461a8880c3134e..d903826894e7f0dfd34dc0caad92eea3caa45e05 100644
--- a/gcc/gimple-pretty-print.cc
+++ b/gcc/gimple-pretty-print.cc
@@ -459,10 +459,6 @@ dump_binary_rhs (pretty_printer *buffer, const gassign *gs, int spc,
case VEC_PACK_FLOAT_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_SERIES_EXPR:
for (p = get_tree_code_name (code); *p; p++)
pp_character (buffer, TOUPPER (*p));
diff --git a/gcc/gimple-range-op.cc b/gcc/gimple-range-op.cc
index 4ca32a7b5d52f8426b09d1446a336650e143b41f..5ae7f7596c6fc6f901e4e47ae44f00185f4602b2 100644
--- a/gcc/gimple-range-op.cc
+++ b/gcc/gimple-range-op.cc
@@ -797,12 +797,6 @@ gimple_range_op_handler::maybe_non_standard ()
if (gimple_code (m_stmt) == GIMPLE_ASSIGN)
switch (gimple_assign_rhs_code (m_stmt))
{
- case WIDEN_PLUS_EXPR:
- {
- signed_op = ptr_op_widen_plus_signed;
- unsigned_op = ptr_op_widen_plus_unsigned;
- }
- gcc_fallthrough ();
case WIDEN_MULT_EXPR:
{
m_valid = false;
diff --git a/gcc/optabs-tree.cc b/gcc/optabs-tree.cc
index 8010046c6a8b3e809c989ddef7a06ddaa68ae32a..ee1aa8c9676ee9c67edbf403e6295da391826a62 100644
--- a/gcc/optabs-tree.cc
+++ b/gcc/optabs-tree.cc
@@ -190,22 +190,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
return (TYPE_UNSIGNED (type)
? vec_widen_ushiftl_lo_optab : vec_widen_sshiftl_lo_optab);
- case VEC_WIDEN_PLUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_lo_optab : vec_widen_saddl_lo_optab);
-
- case VEC_WIDEN_PLUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_hi_optab : vec_widen_saddl_hi_optab);
-
- case VEC_WIDEN_MINUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_lo_optab : vec_widen_ssubl_lo_optab);
-
- case VEC_WIDEN_MINUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_hi_optab : vec_widen_ssubl_hi_optab);
-
case VEC_UNPACK_HI_EXPR:
return (TYPE_UNSIGNED (type)
? vec_unpacku_hi_optab : vec_unpacks_hi_optab);
@@ -312,8 +296,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
'hi'/'lo' pair using codes such as VEC_WIDEN_MINUS_HI/LO.
Supported widening operations:
- WIDEN_MINUS_EXPR
- WIDEN_PLUS_EXPR
WIDEN_MULT_EXPR
WIDEN_LSHIFT_EXPR
@@ -345,12 +327,6 @@ supportable_half_widening_operation (enum tree_code code, tree vectype_out,
case WIDEN_LSHIFT_EXPR:
*code1 = LSHIFT_EXPR;
break;
- case WIDEN_MINUS_EXPR:
- *code1 = MINUS_EXPR;
- break;
- case WIDEN_PLUS_EXPR:
- *code1 = PLUS_EXPR;
- break;
case WIDEN_MULT_EXPR:
*code1 = MULT_EXPR;
break;
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index a9fcc7fd050f871437ef336ecfb8d6cc81280ee0..f80cd1465df83b5540492e619e56b9af249e9f31 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -4017,8 +4017,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case PLUS_EXPR:
case MINUS_EXPR:
{
@@ -4139,10 +4137,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index c702f0032a19203a7c536a01c1e7f47fc7b77add..6e5fd45a0c2435109dd3d50e8fc8e1d4969a1fd0 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4273,8 +4273,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case REALIGN_LOAD_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case DOT_PROD_EXPR:
@@ -4283,10 +4281,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case WIDEN_MULT_MINUS_EXPR:
case WIDEN_LSHIFT_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-pretty-print.cc b/gcc/tree-pretty-print.cc
index 7947f9647a15110b52d195643ad7d28ee32d4236..9941d8bf80535a98e647b8928619a6bf08bc434c 100644
--- a/gcc/tree-pretty-print.cc
+++ b/gcc/tree-pretty-print.cc
@@ -2874,8 +2874,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
break;
/* Binary arithmetic and logic expressions. */
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case MULT_EXPR:
@@ -3831,10 +3829,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
case VEC_SERIES_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
@@ -4352,12 +4346,6 @@ op_symbol_code (enum tree_code code)
case WIDEN_LSHIFT_EXPR:
return "w<<";
- case WIDEN_PLUS_EXPR:
- return "w+";
-
- case WIDEN_MINUS_EXPR:
- return "w-";
-
case POINTER_PLUS_EXPR:
return "+";
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 8daf7bd7dd34d043b1d7b4cba1779f0ecf9f520a..213a3899a6c145bb057cd118bec1df7a05728aef 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -136,8 +136,6 @@ vect_get_smallest_scalar_type (stmt_vec_info stmt_info, tree scalar_type)
|| gimple_assign_rhs_code (assign) == WIDEN_SUM_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_MULT_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_LSHIFT_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_PLUS_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_MINUS_EXPR
|| gimple_assign_rhs_code (assign) == FLOAT_EXPR)
{
tree rhs_type = TREE_TYPE (gimple_assign_rhs1 (assign));
diff --git a/gcc/tree-vect-generic.cc b/gcc/tree-vect-generic.cc
index 445da53292e9d1d2db62ca962fc017bb0e6c9bbe..342ffc5fa7f3b8f37e6bd4658d2f1fccf1d2c7fa 100644
--- a/gcc/tree-vect-generic.cc
+++ b/gcc/tree-vect-generic.cc
@@ -2227,10 +2227,6 @@ expand_vector_operations_1 (gimple_stmt_iterator *gsi,
arguments, not the widened result. VEC_UNPACK_FLOAT_*_EXPR is
calculated in the same way above. */
if (code == WIDEN_SUM_EXPR
- || code == VEC_WIDEN_PLUS_HI_EXPR
- || code == VEC_WIDEN_PLUS_LO_EXPR
- || code == VEC_WIDEN_MINUS_HI_EXPR
- || code == VEC_WIDEN_MINUS_LO_EXPR
|| code == VEC_WIDEN_MULT_HI_EXPR
|| code == VEC_WIDEN_MULT_LO_EXPR
|| code == VEC_WIDEN_MULT_EVEN_EXPR
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 3175dd92187c0935f78ebbf2eb476bdcf8b4ccd1..ab3162b5ac66ea8a96c0ea7c45138ca5ee13423f 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -561,21 +561,30 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
+ return 0;
+
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else if (is_gimple_call (stmt))
+ rhs_code = gimple_call_combined_fn (stmt);
+ else
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ if (rhs_code != code
+ && rhs_code != widened_code)
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = gimple_get_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -588,7 +597,7 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op = gimple_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1342,8 +1351,9 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
- false, 2, unprom, &half_type))
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ CFN_VEC_WIDEN_MINUS, false, 2, unprom,
+ &half_type))
return NULL;
vect_pattern_detected ("vect_recog_sad_pattern", last_stmt);
@@ -2696,9 +2706,10 @@ vect_recog_average_pattern (vec_info *vinfo,
internal_fn ifn = IFN_AVG_FLOOR;
vect_unpromoted_value unprom[3];
tree new_type;
+ enum optab_subtype subtype;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
- unprom, &new_type);
+ CFN_VEC_WIDEN_PLUS, false, 3,
+ unprom, &new_type, &subtype);
if (nops == 0)
return NULL;
if (nops == 3)
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 2a7ef2439e12d1966e8884433963a3d387a856b7..ef3ac551f7fe247893b021d98e43c581e2078dbb 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5032,9 +5032,7 @@ vectorizable_conversion (vec_info *vinfo,
else
return false;
- bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
+ bool widen_arith = (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == IFN_VEC_WIDEN_PLUS
|| code == IFN_VEC_WIDEN_MINUS);
@@ -5086,8 +5084,6 @@ vectorizable_conversion (vec_info *vinfo,
{
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
|| code == IFN_VEC_WIDEN_PLUS
|| code == IFN_VEC_WIDEN_MINUS);
@@ -12192,7 +12188,7 @@ supportable_widening_operation (vec_info *vinfo,
class loop *vect_loop = NULL;
machine_mode vec_mode;
enum insn_code icode1, icode2;
- optab optab1, optab2;
+ optab optab1 = unknown_optab, optab2 = unknown_optab;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
tree_code c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
@@ -12290,16 +12286,6 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_WIDEN_LSHIFT_HI_EXPR;
break;
- case WIDEN_PLUS_EXPR:
- c1 = VEC_WIDEN_PLUS_LO_EXPR;
- c2 = VEC_WIDEN_PLUS_HI_EXPR;
- break;
-
- case WIDEN_MINUS_EXPR:
- c1 = VEC_WIDEN_MINUS_LO_EXPR;
- c2 = VEC_WIDEN_MINUS_HI_EXPR;
- break;
-
CASE_CONVERT:
c1 = VEC_UNPACK_LO_EXPR;
c2 = VEC_UNPACK_HI_EXPR;
diff --git a/gcc/tree.def b/gcc/tree.def
index ee02754354f015a16737c7e879d89c3e3be0d5aa..a58e608a90078818a7ade9d1173ac7ec84c48c7a 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1374,15 +1374,16 @@ DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr", tcc_expression, 3)
DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr", tcc_binary, 2)
/* Widening sad (sum of absolute differences).
- The first two arguments are of type t1 which should be integer.
- The third argument and the result are of type t2, such that t2 is at least
- twice the size of t1. Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
+ The first two arguments are of type t1 which should be a vector of integers.
+ The third argument and the result are of type t2, such that the size of
+ the elements of t2 is at least twice the size of the elements of t1.
+ Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
equivalent to:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = PLUS_EXPR (tmp2, arg3)
or:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = WIDEN_SUM_EXPR (tmp2, arg3)
*/
@@ -1421,8 +1422,6 @@ DEFTREECODE (WIDEN_MULT_MINUS_EXPR, "widen_mult_minus_expr", tcc_expression, 3)
the first argument from type t1 to type t2, and then shifting it
by the second argument. */
DEFTREECODE (WIDEN_LSHIFT_EXPR, "widen_lshift_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_PLUS_EXPR, "widen_plus_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_MINUS_EXPR, "widen_minus_expr", tcc_binary, 2)
/* Widening vector multiplication.
The two operands are vectors with N elements of size S. Multiplying the
@@ -1487,10 +1486,6 @@ DEFTREECODE (VEC_PACK_FLOAT_EXPR, "vec_pack_float_expr", tcc_binary, 2)
*/
DEFTREECODE (VEC_WIDEN_LSHIFT_HI_EXPR, "widen_lshift_hi_expr", tcc_binary, 2)
DEFTREECODE (VEC_WIDEN_LSHIFT_LO_EXPR, "widen_lshift_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_HI_EXPR, "widen_plus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_LO_EXPR, "widen_plus_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_HI_EXPR, "widen_minus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_LO_EXPR, "widen_minus_lo_expr", tcc_binary, 2)
/* PREDICT_EXPR. Specify hint for branch prediction. The
PREDICT_EXPR_PREDICTOR specify predictor and PREDICT_EXPR_OUTCOME the
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns
2023-04-25 12:30 ` Richard Biener
@ 2023-04-28 16:06 ` Andre Vieira (lists)
0 siblings, 0 replies; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-04-28 16:06 UTC (permalink / raw)
To: Richard Biener, Richard Sandiford; +Cc: Richard Biener, gcc-patches
On 25/04/2023 13:30, Richard Biener wrote:
> On Mon, 24 Apr 2023, Richard Sandiford wrote:
>
>> Richard Biener <richard.guenther@gmail.com> writes:
>>> On Thu, Apr 20, 2023 at 3:24?PM Andre Vieira (lists) via Gcc-patches
>>> <gcc-patches@gcc.gnu.org> wrote:
>>>>
>>>> Rebased all three patches and made some small changes to the second one:
>>>> - removed sub and abd optabs from commutative_optab_p, I suspect this
>>>> was a copy paste mistake,
>>>> - removed what I believe to be a superfluous switch case in vectorizable
>>>> conversion, the one that was here:
>>>> + if (code.is_fn_code ())
>>>> + {
>>>> + internal_fn ifn = as_internal_fn (code.as_fn_code ());
>>>> + int ecf_flags = internal_fn_flags (ifn);
>>>> + gcc_assert (ecf_flags & ECF_MULTI);
>>>> +
>>>> + switch (code.as_fn_code ())
>>>> + {
>>>> + case CFN_VEC_WIDEN_PLUS:
>>>> + break;
>>>> + case CFN_VEC_WIDEN_MINUS:
>>>> + break;
>>>> + case CFN_LAST:
>>>> + default:
>>>> + return false;
>>>> + }
>>>> +
>>>> + internal_fn lo, hi;
>>>> + lookup_multi_internal_fn (ifn, &lo, &hi);
>>>> + *code1 = as_combined_fn (lo);
>>>> + *code2 = as_combined_fn (hi);
>>>> + optab1 = lookup_multi_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
>>>> + optab2 = lookup_multi_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
>>>> }
>>>>
>>>> I don't think we need to check they are a specfic fn code, as we look-up
>>>> optabs and if they succeed then surely we can vectorize?
>>>>
>>>> OK for trunk?
>>>
>>> In the first patch I see some uses of safe_as_tree_code like
>>>
>>> + if (ch.is_tree_code ())
>>> + return op1 == NULL_TREE ? gimple_build_assign (lhs,
>>> ch.safe_as_tree_code (),
>>> + op0) :
>>> + gimple_build_assign (lhs, ch.safe_as_tree_code (),
>>> + op0, op1);
>>> + else
>>> + {
>>> + internal_fn fn = as_internal_fn (ch.safe_as_fn_code ());
>>> + gimple* stmt;
>>>
>>> where the context actually requires a valid tree code. Please change those
>>> to force to tree code / ifn code. Just use explicit casts here and the other
>>> places that are similar. Before the as_internal_fn just put a
>>> gcc_assert (ch.is_internal_fn ()).
>>
>> Also, doesn't the above ?: simplify to the "else" arm? Null trailing
>> arguments would be ignored for unary operators.
>>
>> I wasn't sure what to make of the op0 handling:
>>
>>> +/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
>>> + or internal_fn contained in ch, respectively. */
>>> +gimple *
>>> +vect_gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
>>> +{
>>> + if (op0 == NULL_TREE)
>>> + return NULL;
>>
>> Can that happen, and if so, does returning null make sense?
>> Maybe an assert would be safer.
>
> Yeah, I was hoping to have a look whether the new gimple_build
> overloads could be used to make this all better (but hoped we can
> finally get this series in in some way).
>
> Richard.
Yeah, in the newest version of the first patch of the series I found
that most of the time I can get away with only really needing to
distinguish between tree_code and internal_fn when building gimple, for
which it currently uses vect_gimple_build, but it does feel like that
could easily be a gimple function.
Having said that, as I partially mention in the patch, I didn't rewrite
the optabs-tree supportable_half_widening and supportable_conversion (or
whatever they are called) because those also at some point need to
access the stmt and there is a massive difference in how we handle
gassigns and gcall's from that perspective, but maybe we can generalize
that too somehow...
Anyway have a look at the new versions (posted just some minutes after
the email I'm replying too haha! timing :P)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 1/3] Refactor to allow internal_fn's
2023-04-28 12:36 ` [PATCH 1/3] Refactor to allow internal_fn's Andre Vieira (lists)
@ 2023-05-03 11:55 ` Richard Biener
2023-05-04 15:20 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-05-03 11:55 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, Richard Sandiford, gcc-patches
On Fri, 28 Apr 2023, Andre Vieira (lists) wrote:
> Hi,
>
> I'm posting the patches separately now with ChangeLogs.
>
> I made the suggested changes and tried to simplify the code a bit further.
> Where internal to tree-vect-stmts I changed most functions to use code_helper
> to avoid having to check at places we didn't need to. I was trying to simplify
> things further by also modifying supportable_half_widening_operation and
> supportable_convert_operation but the result of that was that I ended up
> moving the code to cast to tree code inside them rather than at the call site
> and it didn't look simpler, so I left those. Though if we did make those
> changes we'd no longer need to keep around the tc1 variable in
> vectorizable_conversion... Let me know what you think.
I see that
- else if (CONVERT_EXPR_CODE_P (code)
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
is convenient (as much as I dislike safe_as_tree_code). Isn't
the following
- if (!CONVERT_EXPR_CODE_P (code))
+ if (!CONVERT_EXPR_CODE_P ((tree_code) code))
return false;
then wrong? In other places you added an assert - I assume
that we might want to checking assert in the cast operators?
(those were added mainly for convenience, maybe we want
as_a <>, etc. here - not sure if those will play well with
enums though). Just suggestions for eventual followups in
this area.
+inline enum tree_code
+code_helper::safe_as_tree_code () const
+{
+ return is_tree_code () ? (tree_code) *this : MAX_TREE_CODES;
+}
+
+inline combined_fn
+code_helper::safe_as_fn_code () const {
+ return is_fn_code () ? (combined_fn) *this : CFN_LAST;
+}
+
newline after the last 'const'. Can you place a comment before
these to explain their intended use? Aka give the case the
code isn't the desired kind a safe value?
The patch is OK with just the last bit fixed.
Thanks,
Richard.
> gcc/ChangeLog:
>
> 2023-04-28 Andre Vieira <andre.simoesdiasvieira@arm.com>
> Joel Hutton <joel.hutton@arm.com>
>
> * tree-vect-patterns.cc (vect_gimple_build): New Function.
> (vect_recog_widen_op_pattern): Refactor to use code_helper.
> * tree-vect-stmts.cc (vect_gen_widened_results_half): Likewise.
> (vect_create_vectorized_demotion_stmts): Likewise.
> (vect_create_vectorized_promotion_stmts): Likewise.
> (vect_create_half_widening_stmts): Likewise.
> (vectorizable_conversion): Likewise.
> (vectorizable_call): Likewise.
> (supportable_widening_operation): Likewise.
> (supportable_narrowing_operation): Likewise.
> (simple_integer_narrowing): Likewise.
> * tree-vectorizer.h (supportable_widening_operation): Likewise.
> (supportable_narrowing_operation): Likewise.
> (vect_gimple_build): New function prototype.
> * tree.h (code_helper::safe_as_tree_code): New function.
> (code_helper::safe_as_fn_code): New function.
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-04-28 12:37 ` [PATCH 2/3] Refactor widen_plus as internal_fn Andre Vieira (lists)
@ 2023-05-03 12:11 ` Richard Biener
2023-05-03 19:07 ` Richard Sandiford
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-05-03 12:11 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, Richard Sandiford, gcc-patches
On Fri, 28 Apr 2023, Andre Vieira (lists) wrote:
> This patch replaces the existing tree_code widen_plus and widen_minus
> patterns with internal_fn versions.
>
> DEF_INTERNAL_OPTAB_HILO_FN is like DEF_INTERNAL_OPTAB_FN except it provides
> convenience wrappers for defining conversions that require a hi/lo split, like
> widening and narrowing operations. Each definition for <NAME> will require an
> optab named <OPTAB> and two other optabs that you specify for signed and
> unsigned. The hi/lo pair is necessary because the widening operations take n
> narrow elements as inputs and return n/2 wide elements as outputs. The 'lo'
> operation operates on the first n/2 elements of input. The 'hi' operation
> operates on the second n/2 elements of input. Defining an internal_fn along
> with hi/lo variations allows a single internal function to be returned from a
> vect_recog function that will later be expanded to hi/lo.
>
> DEF_INTERNAL_OPTAB_HILO_FN is used in internal-fn.def to register a widening
> internal_fn. It is defined differently in different places and internal-fn.def
> is sourced from those places so the parameters given can be reused.
> internal-fn.c: defined to expand to hi/lo signed/unsigned optabs, later
> defined to generate the 'expand_' functions for the hi/lo versions of the fn.
> internal-fn.def: defined to invoke DEF_INTERNAL_OPTAB_FN for the original
> and hi/lo variants of the internal_fn
>
> For example:
> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>addl_hi_<mode> ->
> (u/s)addl2
> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>addl_lo_<mode>
> -> (u/s)addl
>
> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
I'll note that it's interesting we have widen multiplication as
the only existing example where we have both HI/LO and EVEN/ODD cases.
I think we want to share as much of the infrastructure to eventually
support targets doing even/odd (I guess all VLA vector targets will
be even/odd?).
DEF_INTERNAL_OPTAB_HILO_FN also looks to be implicitely directed to
widening operations (otherwise no signed/unsigned variants would be
necessary). What I don't understand is why we need an optab
without _hi/_lo but in that case no signed/unsigned variant?
Looks like all plus, plus_lo and plus_hi are commutative but
only plus is widening?! So is the setup that the vectorizer
doesn't know about the split and uses 'plus' but then the
expander performs the split? It does look a bit awkward here
(the plain 'plus' is just used for the scalar case during
pattern recog it seems).
I'd rather have DEF_INTERNAL_OPTAB_HILO_FN split up, declaring
the hi/lo pairs and the scalar variant separately using
DEF_INTERNAL_FN without expander for that, and having
DEF_INTERNAL_HILO_WIDEN_OPTAB_FN and DEF_INTERNAL_EVENODD_WIDEN_OPTAB_FN
for the signed/unsigned pairs? (if we need that helper at all)
Targets shouldn't need to implement the plain optab (it shouldn't
exist) and the vectorizer should query the hi/lo or even/odd
optabs for support instead.
The vectorizer parts look OK to me, I'd like Richard to chime
in on the optab parts as well.
Thanks,
Richard.
> gcc/ChangeLog:
>
> 2023-04-28 Andre Vieira <andre.simoesdiasvieira@arm.com>
> Joel Hutton <joel.hutton@arm.com>
> Tamar Christina <tamar.christina@arm.com>
>
> * internal-fn.cc (INCLUDE_MAP): Include maps for use in optab
> lookup.
> (DEF_INTERNAL_OPTAB_HILO_FN): Macro to define an internal_fn that
> expands into multiple internal_fns (for widening).
> (ifn_cmp): Function to compare ifn's for sorting/searching.
> (lookup_hilo_ifn_optab): Add lookup function.
> (lookup_hilo_internal_fn): Add lookup function.
> (commutative_binary_fn_p): Add widen_plus fn's.
> (widening_fn_p): New function.
> (decomposes_to_hilo_fn_p): New function.
> * internal-fn.def (DEF_INTERNAL_OPTAB_HILO_FN): Define widening
> plus,minus functions.
> (VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS tree code.
> (VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS tree code.
> * internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
> (lookup_hilo_ifn_optab): Add prototype.
> (lookup_hilo_internal_fn): Likewise.
> (widening_fn_p): Likewise.
> (decomposes_to_hilo_fn_p): Likewise.
> * optabs.cc (commutative_optab_p): Add widening plus, minus optabs.
> * optabs.def (OPTAB_CD): widen add, sub optabs
> * tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
> patterns with a hi/lo split.
> (vect_recog_widen_plus_pattern): Refactor to return
> IFN_VECT_WIDEN_PLUS.
> (vect_recog_widen_minus_pattern): Refactor to return new
> IFN_VEC_WIDEN_MINUS.
> * tree-vect-stmts.cc (vectorizable_conversion): Add widen plus/minus
> ifn
> support.
> (supportable_widening_operation): Add widen plus/minus ifn support.
>
> gcc/testsuite/ChangeLog:
>
> * gcc.target/aarch64/vect-widen-add.c: Test that new
> IFN_VEC_WIDEN_PLUS is being used.
> * gcc.target/aarch64/vect-widen-sub.c: Test that new
> IFN_VEC_WIDEN_MINUS is being used.
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 3/3] Remove widen_plus/minus_expr tree codes
2023-04-28 12:37 ` [PATCH 3/3] Remove widen_plus/minus_expr tree codes Andre Vieira (lists)
@ 2023-05-03 12:29 ` Richard Biener
2023-05-10 9:15 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-05-03 12:29 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, Richard Sandiford, gcc-patches
On Fri, 28 Apr 2023, Andre Vieira (lists) wrote:
> This is a rebase of Joel's previous patch.
>
> This patch removes the old widen plus/minus tree codes which have been
> replaced by internal functions.
I guess that's obvious then. I wonder what we do to internal
fns in debug stmts? Looks like we throw those away and do not
generate debug stmts from calls.
Given you remove handling of the scalar WIDEN_PLUS/MINUS_EXPR
codes everywhere do we want to add checking code the scalar
IFNs do not appear in the IL? For at least some cases there
are corresponding functions handling internal functions that
you could have amended otherwise.
Richard.
> gcc/ChangeLog:
>
> 2023-04-28 Andre Vieira <andre.simoesdiasvieira@arm.com>
> Joel Hutton <joel.hutton@arm.com>
>
> * doc/generic.texi: Remove old tree codes.
> * expr.cc (expand_expr_real_2): Remove old tree code cases.
> * gimple-pretty-print.cc (dump_binary_rhs): Likewise.
> * optabs-tree.cc (optab_for_tree_code): Likewise.
> (supportable_half_widening_operation): Likewise.
> * tree-cfg.cc (verify_gimple_assign_binary): Likewise.
> * tree-inline.cc (estimate_operator_cost): Likewise.
> (op_symbol_code): Likewise.
> * tree-vect-data-refs.cc (vect_get_smallest_scalar_type): Likewise.
> (vect_analyze_data_ref_accesses): Likewise.
> * tree-vect-generic.cc (expand_vector_operations_1): Likewise.
> * cfgexpand.cc (expand_debug_expr): Likewise.
> * tree-vect-stmts.cc (vectorizable_conversion): Likewise.
> (supportable_widening_operation): Likewise.
> * gimple-range-op.cc (gimple_range_op_handler::maybe_non_standard):
> Likewise.
> * tree-vect-patterns.cc (vect_widened_op_tree): Refactor to replace
> usage in vect_recog_sad_pattern.
> (vect_recog_sad_pattern): Replace tree code widening pattern with
> internal function.
> (vect_recog_average_pattern): Likewise.
> * tree-pretty-print.cc (dump_generic_node): Remove tree code
> definition.
> * tree.def (WIDEN_PLUS_EXPR, WIDEN_MINUS_EXPR, VEC_WIDEN_PLUS_HI_EXPR,
> VEC_WIDEN_PLUS_LO_EXPR, VEC_WIDEN_MINUS_HI_EXPR,
> VEC_WIDEN_MINUS_LO_EXPR): Likewise
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-03 12:11 ` Richard Biener
@ 2023-05-03 19:07 ` Richard Sandiford
2023-05-12 12:16 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Sandiford @ 2023-05-03 19:07 UTC (permalink / raw)
To: Richard Biener; +Cc: Andre Vieira (lists), Richard Biener, gcc-patches
Richard Biener <rguenther@suse.de> writes:
> On Fri, 28 Apr 2023, Andre Vieira (lists) wrote:
>
>> This patch replaces the existing tree_code widen_plus and widen_minus
>> patterns with internal_fn versions.
>>
>> DEF_INTERNAL_OPTAB_HILO_FN is like DEF_INTERNAL_OPTAB_FN except it provides
>> convenience wrappers for defining conversions that require a hi/lo split, like
>> widening and narrowing operations. Each definition for <NAME> will require an
>> optab named <OPTAB> and two other optabs that you specify for signed and
>> unsigned. The hi/lo pair is necessary because the widening operations take n
>> narrow elements as inputs and return n/2 wide elements as outputs. The 'lo'
>> operation operates on the first n/2 elements of input. The 'hi' operation
>> operates on the second n/2 elements of input. Defining an internal_fn along
>> with hi/lo variations allows a single internal function to be returned from a
>> vect_recog function that will later be expanded to hi/lo.
>>
>> DEF_INTERNAL_OPTAB_HILO_FN is used in internal-fn.def to register a widening
>> internal_fn. It is defined differently in different places and internal-fn.def
>> is sourced from those places so the parameters given can be reused.
>> internal-fn.c: defined to expand to hi/lo signed/unsigned optabs, later
>> defined to generate the 'expand_' functions for the hi/lo versions of the fn.
>> internal-fn.def: defined to invoke DEF_INTERNAL_OPTAB_FN for the original
>> and hi/lo variants of the internal_fn
>>
>> For example:
>> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
>> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>addl_hi_<mode> ->
>> (u/s)addl2
>> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>addl_lo_<mode>
>> -> (u/s)addl
>>
>> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
>> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
>
> I'll note that it's interesting we have widen multiplication as
> the only existing example where we have both HI/LO and EVEN/ODD cases.
> I think we want to share as much of the infrastructure to eventually
> support targets doing even/odd (I guess all VLA vector targets will
> be even/odd?).
Can't speak for all, but SVE2 certainly is.
> DEF_INTERNAL_OPTAB_HILO_FN also looks to be implicitely directed to
> widening operations (otherwise no signed/unsigned variants would be
> necessary). What I don't understand is why we need an optab
> without _hi/_lo but in that case no signed/unsigned variant?
>
> Looks like all plus, plus_lo and plus_hi are commutative but
> only plus is widening?! So is the setup that the vectorizer
> doesn't know about the split and uses 'plus' but then the
> expander performs the split? It does look a bit awkward here
> (the plain 'plus' is just used for the scalar case during
> pattern recog it seems).
>
> I'd rather have DEF_INTERNAL_OPTAB_HILO_FN split up, declaring
> the hi/lo pairs and the scalar variant separately using
> DEF_INTERNAL_FN without expander for that, and having
> DEF_INTERNAL_HILO_WIDEN_OPTAB_FN and DEF_INTERNAL_EVENODD_WIDEN_OPTAB_FN
> for the signed/unsigned pairs? (if we need that helper at all)
>
> Targets shouldn't need to implement the plain optab (it shouldn't
> exist) and the vectorizer should query the hi/lo or even/odd
> optabs for support instead.
I dread these kinds of review because I think I'm almost certain to
flatly contradict something I said last time round, but +1 FWIW.
It seems OK to define an ifn to represent the combined effect, for the
scalar case, but that shouldn't leak into optabs unless we actually want
to use the ifn for "real" scalar ops (as opposed to a temporary
placeholder during pattern recognition).
On the optabs/ifn bits:
> +static int
> +ifn_cmp (const void *a_, const void *b_)
> +{
> + typedef std::pair<enum internal_fn, unsigned> ifn_pair;
> + auto *a = (const std::pair<ifn_pair, optab> *)a_;
> + auto *b = (const std::pair<ifn_pair, optab> *)b_;
> + return (int) (a->first.first) - (b->first.first);
> +}
> +
> +/* Return the optab belonging to the given internal function NAME for the given
> + SIGN or unknown_optab. */
> +
> +optab
> +lookup_hilo_ifn_optab (enum internal_fn fn, unsigned sign)
There is no NAME parameter. It also isn't clear what SIGN means:
is 1 for unsigned or signed? Would be better to use signop and
TYPE_SIGN IMO.
> +{
> + typedef std::pair<enum internal_fn, unsigned> ifn_pair;
> + typedef auto_vec <std::pair<ifn_pair, optab>>fn_to_optab_map_type;
> + static fn_to_optab_map_type *fn_to_optab_map;
> +
> + if (!fn_to_optab_map)
> + {
> + unsigned num
> + = sizeof (internal_fn_hilo_keys_array) / sizeof (enum internal_fn);
> + fn_to_optab_map = new fn_to_optab_map_type ();
> + for (unsigned int i = 0; i < num - 1; ++i)
> + {
> + enum internal_fn fn = internal_fn_hilo_keys_array[i];
> + optab v1 = internal_fn_hilo_values_array[2*i];
> + optab v2 = internal_fn_hilo_values_array[2*i + 1];
> + ifn_pair key1 (fn, 0);
> + fn_to_optab_map->safe_push ({key1, v1});
> + ifn_pair key2 (fn, 1);
> + fn_to_optab_map->safe_push ({key2, v2});
> + }
> + fn_to_optab_map->qsort (ifn_cmp);
> + }
> +
> + ifn_pair new_pair (fn, sign ? 1 : 0);
> + optab tmp;
> + std::pair<ifn_pair,optab> pair_wrap (new_pair, tmp);
> + auto entry = fn_to_optab_map->bsearch (&pair_wrap, ifn_cmp);
> + return entry != fn_to_optab_map->end () ? entry->second : unknown_optab;
> +}
> +
Do we need to use a map for this? It seems like it follows mechanically
from the macro definition and could be handled using a switch statement
and preprocessor logic.
Also, it would be good to make direct_internal_fn_optab DTRT for this
case, rather than needing a separate function.
> +extern void
> +lookup_hilo_internal_fn (enum internal_fn ifn, enum internal_fn *lo,
> + enum internal_fn *hi)
> +{
> + gcc_assert (decomposes_to_hilo_fn_p (ifn));
> +
> + *lo = internal_fn (ifn + 1);
> + *hi = internal_fn (ifn + 2);
> +}
Nit: spurious extern. Function needs a comment. There have been
requests to drop redundant "enum" keywords from new code.
> +/* Return true if FN decomposes to _hi and _lo IFN. If true this should also
> + be a widening function. */
> +
> +bool
> +decomposes_to_hilo_fn_p (internal_fn fn)
> +{
> + if (!widening_fn_p (fn))
> + return false;
> +
> + switch (fn)
> + {
> + case IFN_VEC_WIDEN_PLUS:
> + case IFN_VEC_WIDEN_MINUS:
> + return true;
> +
> + default:
> + return false;
> + }
> +}
> +
Similarly here I think we should use the preprocessor. It isn't clear
why this returns false for !widening_fn_p. Narrowing hi/lo functions
would decompose in a similar way.
As a general comment, how about naming the new macro:
DEF_INTERNAL_SIGNED_HILO_OPTAB_FN
and make it invoke DEF_INTERNAL_SIGNED_OPTAB_FN twice, once for
the hi and once for the lo?
The new optabs need to be documented in md.texi. I think it'd be
better to drop the "l" suffix in "addl" and "subl", since that's an
Arm convention and is redundant with the earlier "widen".
Sorry for the nitpicks and thanks for picking up this work.
Richard
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 1/3] Refactor to allow internal_fn's
2023-05-03 11:55 ` Richard Biener
@ 2023-05-04 15:20 ` Andre Vieira (lists)
2023-05-05 6:09 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-04 15:20 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
On 03/05/2023 12:55, Richard Biener wrote:
> On Fri, 28 Apr 2023, Andre Vieira (lists) wrote:
>
>> Hi,
>>
>> I'm posting the patches separately now with ChangeLogs.
>>
>> I made the suggested changes and tried to simplify the code a bit further.
>> Where internal to tree-vect-stmts I changed most functions to use code_helper
>> to avoid having to check at places we didn't need to. I was trying to simplify
>> things further by also modifying supportable_half_widening_operation and
>> supportable_convert_operation but the result of that was that I ended up
>> moving the code to cast to tree code inside them rather than at the call site
>> and it didn't look simpler, so I left those. Though if we did make those
>> changes we'd no longer need to keep around the tc1 variable in
>> vectorizable_conversion... Let me know what you think.
>
> I see that
>
> - else if (CONVERT_EXPR_CODE_P (code)
> + else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
>
> is convenient (as much as I dislike safe_as_tree_code). Isn't
> the following
>
> - if (!CONVERT_EXPR_CODE_P (code))
> + if (!CONVERT_EXPR_CODE_P ((tree_code) code))
> return false;
For some reason I thought the code could only reach here if code was a
tree code, but I guess if we have an ifn and the modes aren't the same
as the wide_vectype it would fall to this, which for an ifn this would
fail. I am wondering whether it needs to though, the multi-step widening
should also work for ifn's no? We'd need to adapt it, to not use c1, c2
but hi, lo in case of ifn I guess.. and then use a different optab look
up too?
Though I'm thinking, maybe this should be a follow-up and just not have
that 'feature' for now. The feature being, supporting multi-step
conversion for new widening IFN's.
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 1/3] Refactor to allow internal_fn's
2023-05-04 15:20 ` Andre Vieira (lists)
@ 2023-05-05 6:09 ` Richard Biener
2023-05-12 12:14 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-05-05 6:09 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, Richard Sandiford, gcc-patches
On Thu, 4 May 2023, Andre Vieira (lists) wrote:
>
>
> On 03/05/2023 12:55, Richard Biener wrote:
> > On Fri, 28 Apr 2023, Andre Vieira (lists) wrote:
> >
> >> Hi,
> >>
> >> I'm posting the patches separately now with ChangeLogs.
> >>
> >> I made the suggested changes and tried to simplify the code a bit further.
> >> Where internal to tree-vect-stmts I changed most functions to use
> >> code_helper
> >> to avoid having to check at places we didn't need to. I was trying to
> >> simplify
> >> things further by also modifying supportable_half_widening_operation and
> >> supportable_convert_operation but the result of that was that I ended up
> >> moving the code to cast to tree code inside them rather than at the call
> >> site
> >> and it didn't look simpler, so I left those. Though if we did make those
> >> changes we'd no longer need to keep around the tc1 variable in
> >> vectorizable_conversion... Let me know what you think.
> >
> > I see that
> >
> > - else if (CONVERT_EXPR_CODE_P (code)
> > + else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
> >
> > is convenient (as much as I dislike safe_as_tree_code). Isn't
> > the following
> >
> > - if (!CONVERT_EXPR_CODE_P (code))
> > + if (!CONVERT_EXPR_CODE_P ((tree_code) code))
> > return false;
> For some reason I thought the code could only reach here if code was a tree
> code, but I guess if we have an ifn and the modes aren't the same as the
> wide_vectype it would fall to this, which for an ifn this would fail. I am
> wondering whether it needs to though, the multi-step widening should also work
> for ifn's no? We'd need to adapt it, to not use c1, c2 but hi, lo in case of
> ifn I guess.. and then use a different optab look up too?
>
> Though I'm thinking, maybe this should be a follow-up and just not have that
> 'feature' for now. The feature being, supporting multi-step conversion for new
> widening IFN's.
Yes, I think we should address this in a followup if needed.
Richard.
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 3/3] Remove widen_plus/minus_expr tree codes
2023-05-03 12:29 ` Richard Biener
@ 2023-05-10 9:15 ` Andre Vieira (lists)
2023-05-12 12:18 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-10 9:15 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
On 03/05/2023 13:29, Richard Biener wrote:
> On Fri, 28 Apr 2023, Andre Vieira (lists) wrote:
>
>> This is a rebase of Joel's previous patch.
>>
>> This patch removes the old widen plus/minus tree codes which have been
>> replaced by internal functions.
>
> I guess that's obvious then. I wonder what we do to internal
> fns in debug stmts? Looks like we throw those away and do not
> generate debug stmts from calls.
See the comment above the removed lines in expand_debug_expr:
/* Vector stuff. For most of the codes we don't have rtl codes. */
And it then just returns NULL for those expr's. So the behaviour there
remains unchanged, not saying we couldn't do anything but I don
>
> Given you remove handling of the scalar WIDEN_PLUS/MINUS_EXPR
> codes everywhere do we want to add checking code the scalar
> IFNs do not appear in the IL? For at least some cases there
> are corresponding functions handling internal functions that
> you could have amended otherwise.
I am making some changes to PATCH 2 of this series, in the new version I
am adding some extra code to the gimple checks, one of which is to error
if it comes a cross an IFN that decomposes to HILO as that should only
occur as an intermediary representation of the vect pass.
>
> Richard.
>
>> gcc/ChangeLog:
>>
>> 2023-04-28 Andre Vieira <andre.simoesdiasvieira@arm.com>
>> Joel Hutton <joel.hutton@arm.com>
>>
>> * doc/generic.texi: Remove old tree codes.
>> * expr.cc (expand_expr_real_2): Remove old tree code cases.
>> * gimple-pretty-print.cc (dump_binary_rhs): Likewise.
>> * optabs-tree.cc (optab_for_tree_code): Likewise.
>> (supportable_half_widening_operation): Likewise.
>> * tree-cfg.cc (verify_gimple_assign_binary): Likewise.
>> * tree-inline.cc (estimate_operator_cost): Likewise.
>> (op_symbol_code): Likewise.
>> * tree-vect-data-refs.cc (vect_get_smallest_scalar_type): Likewise.
>> (vect_analyze_data_ref_accesses): Likewise.
>> * tree-vect-generic.cc (expand_vector_operations_1): Likewise.
>> * cfgexpand.cc (expand_debug_expr): Likewise.
>> * tree-vect-stmts.cc (vectorizable_conversion): Likewise.
>> (supportable_widening_operation): Likewise.
>> * gimple-range-op.cc (gimple_range_op_handler::maybe_non_standard):
>> Likewise.
>> * tree-vect-patterns.cc (vect_widened_op_tree): Refactor to replace
>> usage in vect_recog_sad_pattern.
>> (vect_recog_sad_pattern): Replace tree code widening pattern with
>> internal function.
>> (vect_recog_average_pattern): Likewise.
>> * tree-pretty-print.cc (dump_generic_node): Remove tree code
>> definition.
>> * tree.def (WIDEN_PLUS_EXPR, WIDEN_MINUS_EXPR, VEC_WIDEN_PLUS_HI_EXPR,
>> VEC_WIDEN_PLUS_LO_EXPR, VEC_WIDEN_MINUS_HI_EXPR,
>> VEC_WIDEN_MINUS_LO_EXPR): Likewise
>>
>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 1/3] Refactor to allow internal_fn's
2023-05-05 6:09 ` Richard Biener
@ 2023-05-12 12:14 ` Andre Vieira (lists)
2023-05-12 13:18 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-12 12:14 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]
Hi,
I think I tackled all of your comments, let me know if I missed something.
gcc/ChangeLog:
2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
* tree-vect-patterns.cc (vect_gimple_build): New Function.
(vect_recog_widen_op_pattern): Refactor to use code_helper.
* tree-vect-stmts.cc (vect_gen_widened_results_half): Likewise.
(vect_create_vectorized_demotion_stmts): Likewise.
(vect_create_vectorized_promotion_stmts): Likewise.
(vect_create_half_widening_stmts): Likewise.
(vectorizable_conversion): Likewise.
(vectorizable_call): Likewise.
(supportable_widening_operation): Likewise.
(supportable_narrowing_operation): Likewise.
(simple_integer_narrowing): Likewise.
* tree-vectorizer.h (supportable_widening_operation): Likewise.
(supportable_narrowing_operation): Likewise.
(vect_gimple_build): New function prototype.
* tree.h (code_helper::safe_as_tree_code): New function.
(code_helper::safe_as_fn_code): New function.
[-- Attachment #2: ifn0v3.patch --]
[-- Type: text/plain, Size: 23124 bytes --]
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 33a8b2bb60601dc1a67de62a56bbf3c355e12dbd..1778af0242898e3dc73d94d22a5b8505628a53b5 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -25,6 +25,8 @@ along with GCC; see the file COPYING3. If not see
#include "rtl.h"
#include "tree.h"
#include "gimple.h"
+#include "gimple-iterator.h"
+#include "gimple-fold.h"
#include "ssa.h"
#include "expmed.h"
#include "optabs-tree.h"
@@ -1392,7 +1394,7 @@ vect_recog_sad_pattern (vec_info *vinfo,
static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
- tree_code orig_code, tree_code wide_code,
+ tree_code orig_code, code_helper wide_code,
bool shift_p, const char *name)
{
gimple *last_stmt = last_stmt_info->stmt;
@@ -1435,7 +1437,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
vecctype = get_vectype_for_scalar_type (vinfo, ctype);
}
- enum tree_code dummy_code;
+ code_helper dummy_code;
int dummy_int;
auto_vec<tree> dummy_vec;
if (!vectype
@@ -1456,8 +1458,7 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
2, oprnd, half_type, unprom, vectype);
tree var = vect_recog_temp_ssa_var (itype, NULL);
- gimple *pattern_stmt = gimple_build_assign (var, wide_code,
- oprnd[0], oprnd[1]);
+ gimple *pattern_stmt = vect_gimple_build (var, wide_code, oprnd[0], oprnd[1]);
if (vecctype != vecitype)
pattern_stmt = vect_convert_output (vinfo, last_stmt_info, ctype,
@@ -6808,3 +6809,20 @@ vect_pattern_recog (vec_info *vinfo)
/* After this no more add_stmt calls are allowed. */
vinfo->stmt_vec_info_ro = true;
}
+
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple *
+vect_gimple_build (tree lhs, code_helper ch, tree op0, tree op1)
+{
+ gcc_assert (op0 != NULL_TREE);
+ if (ch.is_tree_code ())
+ return gimple_build_assign (lhs, (tree_code) ch, op0, op1);
+
+ gcc_assert (ch.is_internal_fn ());
+ gimple* stmt = gimple_build_call_internal (as_internal_fn ((combined_fn) ch),
+ op1 == NULL_TREE ? 1 : 2,
+ op0, op1);
+ gimple_call_set_lhs (stmt, lhs);
+ return stmt;
+}
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 61a2da4ecee9c449c1469cab3c4cfa1a782471d5..d152ae9ab10b361b88c0f839d6951c43b954750a 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -3261,13 +3261,13 @@ vectorizable_bswap (vec_info *vinfo,
static bool
simple_integer_narrowing (tree vectype_out, tree vectype_in,
- tree_code *convert_code)
+ code_helper *convert_code)
{
if (!INTEGRAL_TYPE_P (TREE_TYPE (vectype_out))
|| !INTEGRAL_TYPE_P (TREE_TYPE (vectype_in)))
return false;
- tree_code code;
+ code_helper code;
int multi_step_cvt = 0;
auto_vec <tree, 8> interm_types;
if (!supportable_narrowing_operation (NOP_EXPR, vectype_out, vectype_in,
@@ -3481,7 +3481,7 @@ vectorizable_call (vec_info *vinfo,
tree callee = gimple_call_fndecl (stmt);
/* First try using an internal function. */
- tree_code convert_code = ERROR_MARK;
+ code_helper convert_code = MAX_TREE_CODES;
if (cfn != CFN_LAST
&& (modifier == NONE
|| (modifier == NARROW
@@ -3667,8 +3667,8 @@ vectorizable_call (vec_info *vinfo,
continue;
}
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, convert_code,
- prev_res, half_res);
+ new_stmt = vect_gimple_build (new_temp, convert_code,
+ prev_res, half_res);
vect_finish_stmt_generation (vinfo, stmt_info,
new_stmt, gsi);
}
@@ -3758,8 +3758,8 @@ vectorizable_call (vec_info *vinfo,
continue;
}
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, convert_code,
- prev_res, half_res);
+ new_stmt = vect_gimple_build (new_temp, convert_code, prev_res,
+ half_res);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -4771,7 +4771,7 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info,
STMT_INFO is the original scalar stmt that we are vectorizing. */
static gimple *
-vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
+vect_gen_widened_results_half (vec_info *vinfo, code_helper ch,
tree vec_oprnd0, tree vec_oprnd1, int op_type,
tree vec_dest, gimple_stmt_iterator *gsi,
stmt_vec_info stmt_info)
@@ -4780,12 +4780,11 @@ vect_gen_widened_results_half (vec_info *vinfo, enum tree_code code,
tree new_temp;
/* Generate half of the widened result: */
- gcc_assert (op_type == TREE_CODE_LENGTH (code));
if (op_type != binary_op)
vec_oprnd1 = NULL;
- new_stmt = gimple_build_assign (vec_dest, code, vec_oprnd0, vec_oprnd1);
+ new_stmt = vect_gimple_build (vec_dest, ch, vec_oprnd0, vec_oprnd1);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
return new_stmt;
@@ -4802,7 +4801,7 @@ vect_create_vectorized_demotion_stmts (vec_info *vinfo, vec<tree> *vec_oprnds,
stmt_vec_info stmt_info,
vec<tree> &vec_dsts,
gimple_stmt_iterator *gsi,
- slp_tree slp_node, enum tree_code code)
+ slp_tree slp_node, code_helper code)
{
unsigned int i;
tree vop0, vop1, new_tmp, vec_dest;
@@ -4814,9 +4813,9 @@ vect_create_vectorized_demotion_stmts (vec_info *vinfo, vec<tree> *vec_oprnds,
/* Create demotion operation. */
vop0 = (*vec_oprnds)[i];
vop1 = (*vec_oprnds)[i + 1];
- gassign *new_stmt = gimple_build_assign (vec_dest, code, vop0, vop1);
+ gimple *new_stmt = vect_gimple_build (vec_dest, code, vop0, vop1);
new_tmp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_tmp);
+ gimple_set_lhs (new_stmt, new_tmp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
if (multi_step_cvt)
@@ -4864,8 +4863,8 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
- enum tree_code code2, int op_type)
+ code_helper ch1,
+ code_helper ch2, int op_type)
{
int i;
tree vop0, vop1, new_tmp1, new_tmp2;
@@ -4881,10 +4880,10 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo,
vop1 = NULL_TREE;
/* Generate the two halves of promotion operation. */
- new_stmt1 = vect_gen_widened_results_half (vinfo, code1, vop0, vop1,
+ new_stmt1 = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
- new_stmt2 = vect_gen_widened_results_half (vinfo, code2, vop0, vop1,
+ new_stmt2 = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1,
op_type, vec_dest, gsi,
stmt_info);
if (is_gimple_call (new_stmt1))
@@ -4915,7 +4914,7 @@ vect_create_half_widening_stmts (vec_info *vinfo,
vec<tree> *vec_oprnds1,
stmt_vec_info stmt_info, tree vec_dest,
gimple_stmt_iterator *gsi,
- enum tree_code code1,
+ code_helper code1,
int op_type)
{
int i;
@@ -4945,13 +4944,13 @@ vect_create_half_widening_stmts (vec_info *vinfo,
new_stmt2 = gimple_build_assign (new_tmp2, NOP_EXPR, vop1);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt2, gsi);
/* Perform the operation. With both vector inputs widened. */
- new_stmt3 = gimple_build_assign (vec_dest, code1, new_tmp1, new_tmp2);
+ new_stmt3 = vect_gimple_build (vec_dest, code1, new_tmp1, new_tmp2);
}
else
{
/* Perform the operation. With the single vector input widened. */
- new_stmt3 = gimple_build_assign (vec_dest, code1, new_tmp1, vop1);
- }
+ new_stmt3 = vect_gimple_build (vec_dest, code1, new_tmp1, vop1);
+ }
new_tmp3 = make_ssa_name (vec_dest, new_stmt3);
gimple_assign_set_lhs (new_stmt3, new_tmp3);
@@ -4981,8 +4980,9 @@ vectorizable_conversion (vec_info *vinfo,
tree scalar_dest;
tree op0, op1 = NULL_TREE;
loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
- enum tree_code code, code1 = ERROR_MARK, code2 = ERROR_MARK;
- enum tree_code codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
+ tree_code tc1;
+ code_helper code, code1, code2;
+ code_helper codecvt1 = ERROR_MARK, codecvt2 = ERROR_MARK;
tree new_temp;
enum vect_def_type dt[2] = {vect_unknown_def_type, vect_unknown_def_type};
int ndts = 2;
@@ -5011,31 +5011,43 @@ vectorizable_conversion (vec_info *vinfo,
&& ! vec_stmt)
return false;
- gassign *stmt = dyn_cast <gassign *> (stmt_info->stmt);
- if (!stmt)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
return false;
- if (TREE_CODE (gimple_assign_lhs (stmt)) != SSA_NAME)
+ if (gimple_get_lhs (stmt) == NULL_TREE
+ || TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
return false;
- code = gimple_assign_rhs_code (stmt);
- if (!CONVERT_EXPR_CODE_P (code)
- && code != FIX_TRUNC_EXPR
- && code != FLOAT_EXPR
- && code != WIDEN_PLUS_EXPR
- && code != WIDEN_MINUS_EXPR
- && code != WIDEN_MULT_EXPR
- && code != WIDEN_LSHIFT_EXPR)
+ if (TREE_CODE (gimple_get_lhs (stmt)) != SSA_NAME)
+ return false;
+
+ if (is_gimple_assign (stmt))
+ {
+ code = gimple_assign_rhs_code (stmt);
+ op_type = TREE_CODE_LENGTH ((tree_code) code);
+ }
+ else if (gimple_call_internal_p (stmt))
+ {
+ code = gimple_call_internal_fn (stmt);
+ op_type = gimple_call_num_args (stmt);
+ }
+ else
return false;
bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
- op_type = TREE_CODE_LENGTH (code);
+ || code == WIDEN_MINUS_EXPR
+ || code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR);
+
+ if (!widen_arith
+ && !CONVERT_EXPR_CODE_P (code)
+ && code != FIX_TRUNC_EXPR
+ && code != FLOAT_EXPR)
+ return false;
/* Check types of lhs and rhs. */
- scalar_dest = gimple_assign_lhs (stmt);
+ scalar_dest = gimple_get_lhs (stmt);
lhs_type = TREE_TYPE (scalar_dest);
vectype_out = STMT_VINFO_VECTYPE (stmt_info);
@@ -5073,10 +5085,14 @@ vectorizable_conversion (vec_info *vinfo,
if (op_type == binary_op)
{
- gcc_assert (code == WIDEN_MULT_EXPR || code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR || code == WIDEN_MINUS_EXPR);
+ gcc_assert (code == WIDEN_MULT_EXPR
+ || code == WIDEN_LSHIFT_EXPR
+ || code == WIDEN_PLUS_EXPR
+ || code == WIDEN_MINUS_EXPR);
+
- op1 = gimple_assign_rhs2 (stmt);
+ op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
+ gimple_call_arg (stmt, 0);
tree vectype1_in;
if (!vect_is_simple_use (vinfo, stmt_info, slp_node, 1,
&op1, &slp_op1, &dt[1], &vectype1_in))
@@ -5160,8 +5176,13 @@ vectorizable_conversion (vec_info *vinfo,
&& code != FLOAT_EXPR
&& !CONVERT_EXPR_CODE_P (code))
return false;
- if (supportable_convert_operation (code, vectype_out, vectype_in, &code1))
+ gcc_assert (code.is_tree_code ());
+ if (supportable_convert_operation ((tree_code) code, vectype_out,
+ vectype_in, &tc1))
+ {
+ code1 = tc1;
break;
+ }
/* FALLTHRU */
unsupported:
if (dump_enabled_p ())
@@ -5172,9 +5193,12 @@ vectorizable_conversion (vec_info *vinfo,
case WIDEN:
if (known_eq (nunits_in, nunits_out))
{
- if (!supportable_half_widening_operation (code, vectype_out,
- vectype_in, &code1))
+ if (!(code.is_tree_code ()
+ && supportable_half_widening_operation ((tree_code) code,
+ vectype_out, vectype_in,
+ &tc1)))
goto unsupported;
+ code1 = tc1;
gcc_assert (!(multi_step_cvt && op_type == binary_op));
break;
}
@@ -5208,14 +5232,17 @@ vectorizable_conversion (vec_info *vinfo,
if (GET_MODE_SIZE (rhs_mode) == fltsz)
{
- if (!supportable_convert_operation (code, vectype_out,
- cvt_type, &codecvt1))
+ tc1 = ERROR_MARK;
+ gcc_assert (code.is_tree_code ());
+ if (!supportable_convert_operation ((tree_code) code, vectype_out,
+ cvt_type, &tc1))
goto unsupported;
+ codecvt1 = tc1;
}
- else if (!supportable_widening_operation (vinfo, code, stmt_info,
- vectype_out, cvt_type,
- &codecvt1, &codecvt2,
- &multi_step_cvt,
+ else if (!supportable_widening_operation (vinfo, code,
+ stmt_info, vectype_out,
+ cvt_type, &codecvt1,
+ &codecvt2, &multi_step_cvt,
&interm_types))
continue;
else
@@ -5223,8 +5250,9 @@ vectorizable_conversion (vec_info *vinfo,
if (supportable_widening_operation (vinfo, NOP_EXPR, stmt_info,
cvt_type,
- vectype_in, &code1, &code2,
- &multi_step_cvt, &interm_types))
+ vectype_in, &code1,
+ &code2, &multi_step_cvt,
+ &interm_types))
{
found_mode = true;
break;
@@ -5260,9 +5288,11 @@ vectorizable_conversion (vec_info *vinfo,
cvt_type = get_same_sized_vectype (cvt_type, vectype_in);
if (cvt_type == NULL_TREE)
goto unsupported;
- if (!supportable_convert_operation (code, cvt_type, vectype_in,
- &codecvt1))
+ if (!code.is_tree_code ()
+ || !supportable_convert_operation ((tree_code) code, cvt_type,
+ vectype_in, &tc1))
goto unsupported;
+ codecvt1 = tc1;
if (supportable_narrowing_operation (NOP_EXPR, vectype_out, cvt_type,
&code1, &multi_step_cvt,
&interm_types))
@@ -5380,10 +5410,9 @@ vectorizable_conversion (vec_info *vinfo,
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
/* Arguments are ready, create the new vector stmt. */
- gcc_assert (TREE_CODE_LENGTH (code1) == unary_op);
- gassign *new_stmt = gimple_build_assign (vec_dest, code1, vop0);
+ gimple *new_stmt = vect_gimple_build (vec_dest, code1, vop0);
new_temp = make_ssa_name (vec_dest, new_stmt);
- gimple_assign_set_lhs (new_stmt, new_temp);
+ gimple_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
if (slp_node)
@@ -5413,17 +5442,16 @@ vectorizable_conversion (vec_info *vinfo,
for (i = multi_step_cvt; i >= 0; i--)
{
tree this_dest = vec_dsts[i];
- enum tree_code c1 = code1, c2 = code2;
+ code_helper c1 = code1, c2 = code2;
if (i == 0 && codecvt2 != ERROR_MARK)
{
c1 = codecvt1;
c2 = codecvt2;
}
if (known_eq (nunits_out, nunits_in))
- vect_create_half_widening_stmts (vinfo, &vec_oprnds0,
- &vec_oprnds1, stmt_info,
- this_dest, gsi,
- c1, op_type);
+ vect_create_half_widening_stmts (vinfo, &vec_oprnds0, &vec_oprnds1,
+ stmt_info, this_dest, gsi, c1,
+ op_type);
else
vect_create_vectorized_promotion_stmts (vinfo, &vec_oprnds0,
&vec_oprnds1, stmt_info,
@@ -5436,9 +5464,8 @@ vectorizable_conversion (vec_info *vinfo,
gimple *new_stmt;
if (cvt_type)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- new_stmt = gimple_build_assign (new_temp, codecvt1, vop0);
+ new_stmt = vect_gimple_build (new_temp, codecvt1, vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
else
@@ -5462,10 +5489,8 @@ vectorizable_conversion (vec_info *vinfo,
if (cvt_type)
FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0)
{
- gcc_assert (TREE_CODE_LENGTH (codecvt1) == unary_op);
new_temp = make_ssa_name (vec_dest);
- gassign *new_stmt
- = gimple_build_assign (new_temp, codecvt1, vop0);
+ gimple *new_stmt = vect_gimple_build (new_temp, codecvt1, vop0);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
vec_oprnds0[i] = new_temp;
}
@@ -12294,9 +12319,11 @@ vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype)
bool
supportable_widening_operation (vec_info *vinfo,
- enum tree_code code, stmt_vec_info stmt_info,
+ code_helper code,
+ stmt_vec_info stmt_info,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, enum tree_code *code2,
+ code_helper *code1,
+ code_helper *code2,
int *multi_step_cvt,
vec<tree> *interm_types)
{
@@ -12307,7 +12334,7 @@ supportable_widening_operation (vec_info *vinfo,
optab optab1, optab2;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
- enum tree_code c1, c2;
+ tree_code c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
int i;
tree prev_type, intermediate_type;
machine_mode intermediate_mode, prev_mode;
@@ -12317,8 +12344,12 @@ supportable_widening_operation (vec_info *vinfo,
if (loop_info)
vect_loop = LOOP_VINFO_LOOP (loop_info);
- switch (code)
+ switch (code.safe_as_tree_code ())
{
+ case MAX_TREE_CODES:
+ /* Don't set c1 and c2 if code is not a tree_code. */
+ break;
+
case WIDEN_MULT_EXPR:
/* The result of a vectorized widening operation usually requires
two vectors (because the widened results do not fit into one vector).
@@ -12358,8 +12389,9 @@ supportable_widening_operation (vec_info *vinfo,
&& !nested_in_vect_loop_p (vect_loop, stmt_info)
&& supportable_widening_operation (vinfo, VEC_WIDEN_MULT_EVEN_EXPR,
stmt_info, vectype_out,
- vectype_in, code1, code2,
- multi_step_cvt, interm_types))
+ vectype_in, code1,
+ code2, multi_step_cvt,
+ interm_types))
{
/* Elements in a vector with vect_used_by_reduction property cannot
be reordered if the use chain with this property does not have the
@@ -12435,7 +12467,7 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
}
- else if (CONVERT_EXPR_CODE_P (code)
+ else if (CONVERT_EXPR_CODE_P (code.safe_as_tree_code ())
&& VECTOR_BOOLEAN_TYPE_P (wide_vectype)
&& VECTOR_BOOLEAN_TYPE_P (vectype)
&& TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
@@ -12460,8 +12492,12 @@ supportable_widening_operation (vec_info *vinfo,
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- *code1 = c1;
- *code2 = c2;
+ if (code.is_tree_code ())
+ {
+ *code1 = c1;
+ *code2 = c2;
+ }
+
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
@@ -12482,7 +12518,7 @@ supportable_widening_operation (vec_info *vinfo,
prev_type = vectype;
prev_mode = vec_mode;
- if (!CONVERT_EXPR_CODE_P (code))
+ if (!CONVERT_EXPR_CODE_P (code.safe_as_tree_code ()))
return false;
/* We assume here that there will not be more than MAX_INTERM_CVT_STEPS
@@ -12580,9 +12616,9 @@ supportable_widening_operation (vec_info *vinfo,
narrowing operation (short in the above example). */
bool
-supportable_narrowing_operation (enum tree_code code,
+supportable_narrowing_operation (code_helper code,
tree vectype_out, tree vectype_in,
- enum tree_code *code1, int *multi_step_cvt,
+ code_helper *code1, int *multi_step_cvt,
vec<tree> *interm_types)
{
machine_mode vec_mode;
@@ -12597,8 +12633,11 @@ supportable_narrowing_operation (enum tree_code code,
unsigned HOST_WIDE_INT n_elts;
bool uns;
+ if (!code.is_tree_code ())
+ return false;
+
*multi_step_cvt = 0;
- switch (code)
+ switch ((tree_code) code)
{
CASE_CONVERT:
c1 = VEC_PACK_TRUNC_EXPR;
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 9cf2fb23fe397b467d89aa7cc5ebeaa293ed4cce..f215cd0639bcf803c9d0554cfdc57823431991d5 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2139,13 +2139,12 @@ extern bool vect_is_simple_use (vec_info *, stmt_vec_info, slp_tree,
enum vect_def_type *,
tree *, stmt_vec_info * = NULL);
extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree);
-extern bool supportable_widening_operation (vec_info *,
- enum tree_code, stmt_vec_info,
- tree, tree, enum tree_code *,
- enum tree_code *, int *,
- vec<tree> *);
-extern bool supportable_narrowing_operation (enum tree_code, tree, tree,
- enum tree_code *, int *,
+extern bool supportable_widening_operation (vec_info*, code_helper,
+ stmt_vec_info, tree, tree,
+ code_helper*, code_helper*,
+ int*, vec<tree> *);
+extern bool supportable_narrowing_operation (code_helper, tree, tree,
+ code_helper *, int *,
vec<tree> *);
extern unsigned record_stmt_cost (stmt_vector_for_cost *, int,
@@ -2583,4 +2582,7 @@ vect_is_integer_truncation (stmt_vec_info stmt_info)
&& TYPE_PRECISION (lhs_type) < TYPE_PRECISION (rhs_type));
}
+/* Build a GIMPLE_ASSIGN or GIMPLE_CALL with the tree_code,
+ or internal_fn contained in ch, respectively. */
+gimple * vect_gimple_build (tree, code_helper, tree, tree = NULL_TREE);
#endif /* GCC_TREE_VECTORIZER_H */
diff --git a/gcc/tree.h b/gcc/tree.h
index 0b72663e6a1a94406127f6253460f498b7a3ea9c..6dcb28ebc1df456e3798b8f0b43bae42c145d43d 100644
--- a/gcc/tree.h
+++ b/gcc/tree.h
@@ -93,6 +93,8 @@ public:
bool is_internal_fn () const;
bool is_builtin_fn () const;
int get_rep () const { return rep; }
+ tree_code safe_as_tree_code () const;
+ combined_fn safe_as_fn_code () const;
bool operator== (const code_helper &other) { return rep == other.rep; }
bool operator!= (const code_helper &other) { return rep != other.rep; }
bool operator== (tree_code c) { return rep == code_helper (c).rep; }
@@ -102,6 +104,25 @@ private:
int rep;
};
+/* Helper function that returns the tree_code representation of THIS
+ code_helper if it is a tree_code and MAX_TREE_CODES otherwise. This is
+ useful when passing a code_helper to a tree_code only check. */
+
+inline tree_code
+code_helper::safe_as_tree_code () const
+{
+ return is_tree_code () ? (tree_code) *this : MAX_TREE_CODES;
+}
+
+/* Helper function that returns the combined_fn representation of THIS
+ code_helper if it is a fn_code and CFN_LAST otherwise. This is useful when
+ passing a code_helper to a combined_fn only check. */
+
+inline combined_fn
+code_helper::safe_as_fn_code () const {
+ return is_fn_code () ? (combined_fn) *this : CFN_LAST;
+}
+
inline code_helper::operator internal_fn () const
{
return as_internal_fn (combined_fn (*this));
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-03 19:07 ` Richard Sandiford
@ 2023-05-12 12:16 ` Andre Vieira (lists)
2023-05-12 13:28 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-12 12:16 UTC (permalink / raw)
To: Richard Biener, Richard Biener, gcc-patches, richard.sandiford
[-- Attachment #1: Type: text/plain, Size: 4743 bytes --]
I have dealt with, I think..., most of your comments. There's quite a
few changes, I think it's all a bit simpler now. I made some other
changes to the costing in tree-inline.cc and gimple-range-op.cc in which
I try to preserve the same behaviour as we had with the tree codes
before. Also added some extra checks to tree-cfg.cc that made sense to me.
I am still regression testing the gimple-range-op change, as that was a
last minute change, but the rest survived a bootstrap and regression
test on aarch64-unknown-linux-gnu.
cover letter:
This patch replaces the existing tree_code widen_plus and widen_minus
patterns with internal_fn versions.
DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and
DEF_INTERNAL_OPTAB_NARROWING_HILO_FN are like
DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
except they provide convenience wrappers for defining conversions that
require a hi/lo split. Each definition for <NAME> will require optabs
for _hi and _lo and each of those will also require a signed and
unsigned version in the case of widening. The hi/lo pair is necessary
because the widening and narrowing operations take n narrow elements as
inputs and return n/2 wide elements as outputs. The 'lo' operation
operates on the first n/2 elements of input. The 'hi' operation operates
on the second n/2 elements of input. Defining an internal_fn along with
hi/lo variations allows a single internal function to be returned from a
vect_recog function that will later be expanded to hi/lo.
For example:
IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
(u/s)addl2
IFN_VEC_WIDEN_PLUS_LO ->
vec_widen_<su>add_lo_<mode> -> (u/s)addl
This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS
tree codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
gcc/ChangeLog:
2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
Tamar Christina <tamar.christina@arm.com>
* config/aarch64/aarch64-simd.md
(vec_widen_<su>addl_lo_<mode>): Rename
this ...
(vec_widen_<su>add_lo_<mode>): ... to this.
(vec_widen_<su>addl_hi_<mode>): Rename this ...
(vec_widen_<su>add_hi_<mode>): ... to this.
(vec_widen_<su>subl_lo_<mode>): Rename this ...
(vec_widen_<su>sub_lo_<mode>): ... to this.
(vec_widen_<su>subl_hi_<mode>): Rename this ...
(vec_widen_<su>sub_hi_<mode>): ...to this.
* doc/generic.texi: Document new IFN codes.
* internal-fn.cc (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Macro to define an
internal_fn that expands into multiple internal_fns for widening.
(DEF_INTERNAL_OPTAB_NARROWING_HILO_FN): Likewise but for narrowing.
(ifn_cmp): Function to compare ifn's for sorting/searching.
(lookup_hilo_internal_fn): Add lookup function.
(commutative_binary_fn_p): Add widen_plus fn's.
(widening_fn_p): New function.
(narrowing_fn_p): New function.
(decomposes_to_hilo_fn_p): New function.
(direct_internal_fn_optab): Change visibility.
* internal-fn.def (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Define widening
plus,minus functions.
(VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS_EXPR tree code.
(VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS_EXPR tree code.
* internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
(direct_internal_fn_optab): Declare new prototype.
(lookup_hilo_internal_fn): Likewise.
(widening_fn_p): Likewise.
(Narrowing_fn_p): Likewise.
(decomposes_to_hilo_fn_p): Likewise.
* optabs.cc (commutative_optab_p): Add widening plus optabs.
* optabs.def (OPTAB_D): Define widen add, sub optabs.
* tree-cfg.cc (verify_gimple_call): Add checks for new widen
add and sub IFNs.
* tree-inline.cc (estimate_num_insns): Return same
cost for widen add and sub IFNs as previous tree_codes.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
patterns with a hi/lo split.
(vect_recog_sad_pattern): Refactor to use new IFN codes.
(vect_recog_widen_plus_pattern): Likewise.
(vect_recog_widen_minus_pattern): Likewise.
(vect_recog_average_pattern): Likewise.
* tree-vect-stmts.cc (vectorizable_conversion): Add support for
_HILO IFNs.
(supportable_widening_operation): Likewise.
* tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-widen-add.c: Test that new
IFN_VEC_WIDEN_PLUS is being used.
* gcc.target/aarch64/vect-widen-sub.c: Test that new
IFN_VEC_WIDEN_MINUS is being used.
[-- Attachment #2: ifn1v3.patch --]
[-- Type: text/plain, Size: 34240 bytes --]
diff --git a/gcc/config/aarch64/aarch64-simd.md b/gcc/config/aarch64/aarch64-simd.md
index bfc98a8d943467b33390defab9682f44efab5907..ffbbecb9409e1c2835d658c2a8855cd0e955c0f2 100644
--- a/gcc/config/aarch64/aarch64-simd.md
+++ b/gcc/config/aarch64/aarch64-simd.md
@@ -4626,7 +4626,7 @@
[(set_attr "type" "neon_<ADDSUB:optab>_long")]
)
-(define_expand "vec_widen_<su>addl_lo_<mode>"
+(define_expand "vec_widen_<su>add_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4638,7 +4638,7 @@
DONE;
})
-(define_expand "vec_widen_<su>addl_hi_<mode>"
+(define_expand "vec_widen_<su>add_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4650,7 +4650,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_lo_<mode>"
+(define_expand "vec_widen_<su>sub_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4662,7 +4662,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_hi_<mode>"
+(define_expand "vec_widen_<su>sub_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index 8b2882da4fe7da07d22b4e5384d049ba7d3907bf..0fd7e6cce8bbd4ecb8027b702722adcf6c32eb55 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,6 +1811,10 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
+@tindex IFN_VEC_WIDEN_PLUS_HI
+@tindex IFN_VEC_WIDEN_PLUS_LO
+@tindex IFN_VEC_WIDEN_MINUS_HI
+@tindex IFN_VEC_WIDEN_MINUS_LO
@tindex VEC_WIDEN_PLUS_HI_EXPR
@tindex VEC_WIDEN_PLUS_LO_EXPR
@tindex VEC_WIDEN_MINUS_HI_EXPR
@@ -1861,6 +1865,33 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
+@item IFN_VEC_WIDEN_PLUS_HI
+@itemx IFN_VEC_WIDEN_PLUS_LO
+These internal functions represent widening vector addition of the high and low
+parts of the two input vectors, respectively. Their operands are vectors that
+contain the same number of elements (@code{N}) of the same integral type. The
+result is a vector that contains half as many elements, of an integral type
+whose size is twice as wide. In the case of @code{IFN_VEC_WIDEN_PLUS_HI} the
+high @code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} products. In the case of @code{IFN_VEC_WIDEN_PLUS_LO} the low
+@code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} products.
+
+@item IFN_VEC_WIDEN_MINUS_HI
+@itemx IFN_VEC_WIDEN_MINUS_LO
+These internal functions represent widening vector subtraction of the high and
+low parts of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The high/low elements of the second vector are subtracted from the high/low
+elements of the first. The result is a vector that contains half as many
+elements, of an integral type whose size is twice as wide. In the case of
+@code{IFN_VEC_WIDEN_MINUS_HI} the high @code{N/2} elements of the second
+vector are subtracted from the high @code{N/2} of the first to produce the
+vector of @code{N/2} products. In the case of
+@code{IFN_VEC_WIDEN_MINUS_LO} the low @code{N/2} elements of the second
+vector are subtracted from the low @code{N/2} of the first to produce the
+vector of @code{N/2} products.
+
@item VEC_WIDEN_PLUS_HI_EXPR
@itemx VEC_WIDEN_PLUS_LO_EXPR
These nodes represent widening vector addition of the high and low parts of
diff --git a/gcc/gimple-range-op.cc b/gcc/gimple-range-op.cc
index 594bd3043f0e944299ddfff219f757ef15a3dd61..66636d82df27626e7911efd0cb8526921b39633f 100644
--- a/gcc/gimple-range-op.cc
+++ b/gcc/gimple-range-op.cc
@@ -1187,6 +1187,7 @@ gimple_range_op_handler::maybe_non_standard ()
{
range_operator *signed_op = ptr_op_widen_mult_signed;
range_operator *unsigned_op = ptr_op_widen_mult_unsigned;
+ bool signed1, signed2, signed_ret;
if (gimple_code (m_stmt) == GIMPLE_ASSIGN)
switch (gimple_assign_rhs_code (m_stmt))
{
@@ -1202,32 +1203,55 @@ gimple_range_op_handler::maybe_non_standard ()
m_op1 = gimple_assign_rhs1 (m_stmt);
m_op2 = gimple_assign_rhs2 (m_stmt);
tree ret = gimple_assign_lhs (m_stmt);
- bool signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
- bool signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
- bool signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
-
- /* Normally these operands should all have the same sign, but
- some passes and violate this by taking mismatched sign args. At
- the moment the only one that's possible is mismatch inputs and
- unsigned output. Once ranger supports signs for the operands we
- can properly fix it, for now only accept the case we can do
- correctly. */
- if ((signed1 ^ signed2) && signed_ret)
- return;
-
- m_valid = true;
- if (signed2 && !signed1)
- std::swap (m_op1, m_op2);
-
- if (signed1 || signed2)
- m_int = signed_op;
- else
- m_int = unsigned_op;
+ signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
+ signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
+ signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
break;
}
default:
- break;
+ return;
}
+ else if (gimple_code (m_stmt) == GIMPLE_CALL
+ && gimple_call_internal_p (m_stmt)
+ && gimple_get_lhs (m_stmt) != NULL_TREE)
+ switch (gimple_call_internal_fn (m_stmt))
+ {
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
+ {
+ signed_op = ptr_op_widen_plus_signed;
+ unsigned_op = ptr_op_widen_plus_unsigned;
+ m_valid = false;
+ m_op1 = gimple_call_arg (m_stmt, 0);
+ m_op2 = gimple_call_arg (m_stmt, 1);
+ tree ret = gimple_get_lhs (m_stmt);
+ signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
+ signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
+ signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
+ break;
+ }
+ default:
+ return;
+ }
+ else
+ return;
+
+ /* Normally these operands should all have the same sign, but some passes
+ and violate this by taking mismatched sign args. At the moment the only
+ one that's possible is mismatch inputs and unsigned output. Once ranger
+ supports signs for the operands we can properly fix it, for now only
+ accept the case we can do correctly. */
+ if ((signed1 ^ signed2) && signed_ret)
+ return;
+
+ m_valid = true;
+ if (signed2 && !signed1)
+ std::swap (m_op1, m_op2);
+
+ if (signed1 || signed2)
+ m_int = signed_op;
+ else
+ m_int = unsigned_op;
}
// Set up a gimple_range_op_handler for any built in function which can be
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 5c9da73ea11f8060b18dcf513599c9694fa4f2ad..1acea5ae33046b70de247b1688aea874d9956abc 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -90,6 +90,19 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+/* Given an internal_fn IFN that is a HILO function, return its corresponding
+ LO and HI internal_fns. */
+
+extern void
+lookup_hilo_internal_fn (internal_fn ifn, internal_fn *lo, internal_fn *hi)
+{
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -137,7 +150,16 @@ const direct_internal_fn_info direct_internal_fn_array[IFN_LAST + 1] = {
#define DEF_INTERNAL_OPTAB_FN(CODE, FLAGS, OPTAB, TYPE) TYPE##_direct,
#define DEF_INTERNAL_SIGNED_OPTAB_FN(CODE, FLAGS, SELECTOR, SIGNED_OPTAB, \
UNSIGNED_OPTAB, TYPE) TYPE##_direct,
+#undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+#define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(CODE, FLAGS, SELECTOR, SIGNED_OPTAB, \
+ UNSIGNED_OPTAB, TYPE) \
+TYPE##_direct, TYPE##_direct, TYPE##_direct,
+#define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(CODE, FLAGS, OPTAB, TYPE) \
+TYPE##_direct, TYPE##_direct, TYPE##_direct,
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
not_direct
};
@@ -3852,7 +3874,7 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types,
/* Return the optab used by internal function FN. */
-static optab
+optab
direct_internal_fn_optab (internal_fn fn, tree_pair types)
{
switch (fn)
@@ -3971,6 +3993,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS_HILO:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -4044,6 +4069,88 @@ first_commutative_argument (internal_fn fn)
}
}
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as wide as the element size of the input vectors. */
+
+bool
+widening_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME##_HILO:\
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as narrow as the element size of the input vectors. */
+
+bool
+narrowing_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(NAME, F, O, T) \
+ case IFN_##NAME##_HILO:\
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if FN decomposes to _hi and _lo IFN. */
+
+bool
+decomposes_to_hilo_fn_p (internal_fn fn)
+{
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME##_HILO:\
+ return true;
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(NAME, F, O, T) \
+ case IFN_##NAME##_HILO:\
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+
+ default:
+ return false;
+ }
+}
+
/* Return true if IFN_SET_EDOM is supported. */
bool
@@ -4071,7 +4178,33 @@ set_edom_supported_p (void)
optab which_optab = direct_internal_fn_optab (fn, types); \
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
+#define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(CODE, FLAGS, SELECTOR, \
+ SIGNED_OPTAB, UNSIGNED_OPTAB, \
+ TYPE) \
+ static void \
+ expand_##CODE##_HILO (internal_fn fn ATTRIBUTE_UNUSED, \
+ gcall *stmt ATTRIBUTE_UNUSED) \
+ { \
+ gcc_unreachable (); \
+ } \
+ DEF_INTERNAL_SIGNED_OPTAB_FN(CODE##_HI, FLAGS, SELECTOR, SIGNED_OPTAB, \
+ UNSIGNED_OPTAB, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN(CODE##_LO, FLAGS, SELECTOR, SIGNED_OPTAB, \
+ UNSIGNED_OPTAB, TYPE)
+#define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(CODE, FLAGS, OPTAB, TYPE) \
+ static void \
+ expand_##CODE##_HILO (internal_fn fn ATTRIBUTE_UNUSED, \
+ gcall *stmt ATTRIBUTE_UNUSED) \
+ { \
+ gcc_unreachable (); \
+ } \
+ DEF_INTERNAL_OPTAB_FN(CODE##_LO, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN(CODE##_HI, FLAGS, OPTAB, TYPE)
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_FN
+#undef DEF_INTERNAL_SIGNED_OPTAB_FN
+#undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
@@ -4080,6 +4213,7 @@ set_edom_supported_p (void)
where STMT is the statement that performs the call. */
static void (*const internal_fn_expanders[]) (internal_fn, gcall *) = {
+
#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) expand_##CODE,
#include "internal-fn.def"
0
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 7fe742c2ae713e7152ab05cfdfba86e4e0aa3456..012dd323b86dd7cfcc5c13d3a2bb2a453937155d 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -85,6 +85,13 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_SIGNED_OPTAB_HILO_FN is like DEF_INTERNAL_OPTAB_FN except it
+ provides convenience wrappers for defining conversions that require a
+ hi/lo split, like widening and narrowing operations. Each definition
+ for <NAME> will require an optab named <OPTAB> and two other optabs that
+ you specify for signed and unsigned.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -123,6 +130,20 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(NAME, FLAGS, SELECTOR, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_FN (NAME##_HILO, FLAGS | ECF_LEAF, NULL) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _LO, FLAGS, SELECTOR, SOPTAB##_lo, UOPTAB##_lo, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _HI, FLAGS, SELECTOR, SOPTAB##_hi, UOPTAB##_hi, TYPE)
+#endif
+
+#ifndef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+#define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_FN (NAME##_HILO, FLAGS | ECF_LEAF, NULL) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, OPTAB##_lo, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, OPTAB##_hi, TYPE)
+#endif
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -315,6 +336,16 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_OPTAB_WIDENING_HILO_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_sadd, vec_widen_uadd,
+ binary)
+DEF_INTERNAL_OPTAB_WIDENING_HILO_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_ssub, vec_widen_usub,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 08922ed4254898f5fffca3f33973e96ed9ce772f..8ba07d6d1338e75bc5a451d9e403112a608f3ea2 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,8 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern void lookup_hilo_internal_fn (internal_fn, internal_fn *, internal_fn *);
+extern optab direct_internal_fn_optab (internal_fn, tree_pair);
/* Return the ECF_* flags for function FN. */
@@ -210,6 +216,9 @@ extern bool commutative_binary_fn_p (internal_fn);
extern bool commutative_ternary_fn_p (internal_fn);
extern int first_commutative_argument (internal_fn);
extern bool associative_binary_fn_p (internal_fn);
+extern bool widening_fn_p (code_helper);
+extern bool narrowing_fn_p (code_helper);
+extern bool decomposes_to_hilo_fn_p (internal_fn);
extern bool set_edom_supported_p (void);
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index c8e39c82d57a7d726e7da33d247b80f32ec9236c..5a08d91e550b2d92e9572211f811fdba99a33a38 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,15 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab
+ || binoptab == vec_widen_sadd_hi_optab
+ || binoptab == vec_widen_sadd_lo_optab
+ || binoptab == vec_widen_uadd_hi_optab
+ || binoptab == vec_widen_uadd_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 695f5911b300c9ca5737de9be809fa01aabe5e01..16d121722c8c5723d9b164f5a2c616dc7ec143de 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -410,6 +410,10 @@ OPTAB_D (vec_widen_ssubl_hi_optab, "vec_widen_ssubl_hi_$a")
OPTAB_D (vec_widen_ssubl_lo_optab, "vec_widen_ssubl_lo_$a")
OPTAB_D (vec_widen_saddl_hi_optab, "vec_widen_saddl_hi_$a")
OPTAB_D (vec_widen_saddl_lo_optab, "vec_widen_saddl_lo_$a")
+OPTAB_D (vec_widen_ssub_hi_optab, "vec_widen_ssub_hi_$a")
+OPTAB_D (vec_widen_ssub_lo_optab, "vec_widen_ssub_lo_$a")
+OPTAB_D (vec_widen_sadd_hi_optab, "vec_widen_sadd_hi_$a")
+OPTAB_D (vec_widen_sadd_lo_optab, "vec_widen_sadd_lo_$a")
OPTAB_D (vec_widen_sshiftl_hi_optab, "vec_widen_sshiftl_hi_$a")
OPTAB_D (vec_widen_sshiftl_lo_optab, "vec_widen_sshiftl_lo_$a")
OPTAB_D (vec_widen_umult_even_optab, "vec_widen_umult_even_$a")
@@ -422,6 +426,10 @@ OPTAB_D (vec_widen_usubl_hi_optab, "vec_widen_usubl_hi_$a")
OPTAB_D (vec_widen_usubl_lo_optab, "vec_widen_usubl_lo_$a")
OPTAB_D (vec_widen_uaddl_hi_optab, "vec_widen_uaddl_hi_$a")
OPTAB_D (vec_widen_uaddl_lo_optab, "vec_widen_uaddl_lo_$a")
+OPTAB_D (vec_widen_usub_hi_optab, "vec_widen_usub_hi_$a")
+OPTAB_D (vec_widen_usub_lo_optab, "vec_widen_usub_lo_$a")
+OPTAB_D (vec_widen_uadd_hi_optab, "vec_widen_uadd_hi_$a")
+OPTAB_D (vec_widen_uadd_lo_optab, "vec_widen_uadd_lo_$a")
OPTAB_D (vec_addsub_optab, "vec_addsub$a3")
OPTAB_D (vec_fmaddsub_optab, "vec_fmaddsub$a4")
OPTAB_D (vec_fmsubadd_optab, "vec_fmsubadd$a4")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index 0aeebb67fac864db284985f4a6f0653af281d62b..28464ad9e3a7ea25557ffebcdbdbc1340f9e0d8b 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -65,6 +65,7 @@ along with GCC; see the file COPYING3. If not see
#include "asan.h"
#include "profile.h"
#include "sreal.h"
+#include "internal-fn.h"
/* This file contains functions for building the Control Flow Graph (CFG)
for a function tree. */
@@ -3411,6 +3412,52 @@ verify_gimple_call (gcall *stmt)
debug_generic_stmt (fn);
return true;
}
+ internal_fn ifn = gimple_call_internal_fn (stmt);
+ if (ifn == IFN_LAST)
+ {
+ error ("gimple call has an invalid IFN");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ else if (decomposes_to_hilo_fn_p (ifn))
+ {
+ /* Non decomposed HILO stmts should not appear in IL, these are
+ merely used as an internal representation to the auto-vectorizer
+ pass and should have been expanded to their _LO _HI variants. */
+ error ("gimple call has an non decomposed HILO IFN");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ else if (ifn == IFN_VEC_WIDEN_PLUS_LO
+ || ifn == IFN_VEC_WIDEN_PLUS_HI
+ || ifn == IFN_VEC_WIDEN_MINUS_LO
+ || ifn == IFN_VEC_WIDEN_MINUS_HI)
+ {
+ tree rhs1_type = TREE_TYPE (gimple_call_arg (stmt, 0));
+ tree rhs2_type = TREE_TYPE (gimple_call_arg (stmt, 1));
+ tree lhs_type = TREE_TYPE (gimple_get_lhs (stmt));
+ if (TREE_CODE (lhs_type) == VECTOR_TYPE)
+ {
+ if (TREE_CODE (rhs1_type) != VECTOR_TYPE
+ || TREE_CODE (rhs2_type) != VECTOR_TYPE)
+ {
+ error ("invalid non-vector operands in vector IFN call");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ lhs_type = TREE_TYPE (lhs_type);
+ rhs1_type = TREE_TYPE (rhs1_type);
+ rhs2_type = TREE_TYPE (rhs2_type);
+ }
+ if (POINTER_TYPE_P (lhs_type)
+ || POINTER_TYPE_P (rhs1_type)
+ || POINTER_TYPE_P (rhs2_type))
+ {
+ error ("invalid (pointer) operands in vector IFN call");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ }
}
else
{
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 63a19f8d1d89c6bd5d8e55a299cbffaa324b4b84..d74d8db2173b1ab117250fea89de5212d5e354ec 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4433,7 +4433,20 @@ estimate_num_insns (gimple *stmt, eni_weights *weights)
tree decl;
if (gimple_call_internal_p (stmt))
- return 0;
+ {
+ internal_fn fn = gimple_call_internal_fn (stmt);
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS_HI:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_MINUS_HI:
+ case IFN_VEC_WIDEN_MINUS_LO:
+ return 1;
+
+ default:
+ return 0;
+ }
+ }
else if ((decl = gimple_call_fndecl (stmt))
&& fndecl_built_in_p (decl))
{
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 1778af0242898e3dc73d94d22a5b8505628a53b5..93cebc72beb4f65249a69b2665dfeb8a0991c1d1 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -562,21 +562,30 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
+ return 0;
+
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else if (is_gimple_call (stmt))
+ rhs_code = gimple_call_combined_fn (stmt);
+ else
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ if (rhs_code != code
+ && rhs_code != widened_code)
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = gimple_get_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -589,7 +598,7 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op = gimple_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1343,7 +1352,8 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ IFN_VEC_WIDEN_MINUS_HILO,
false, 2, unprom, &half_type))
return NULL;
@@ -1395,14 +1405,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1468,6 +1480,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1481,26 +1507,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS_HILO. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS_HILO,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS_HILO. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS_HILO,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_ctz_ffs_pattern
@@ -3078,7 +3108,7 @@ vect_recog_average_pattern (vec_info *vinfo,
vect_unpromoted_value unprom[3];
tree new_type;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
+ IFN_VEC_WIDEN_PLUS_HILO, false, 3,
unprom, &new_type);
if (nops == 0)
return NULL;
@@ -6469,6 +6499,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index d152ae9ab10b361b88c0f839d6951c43b954750a..24c811ebe01fb8b003100dea494cf64fea72a975 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5038,7 +5038,9 @@ vectorizable_conversion (vec_info *vinfo,
bool widen_arith = (code == WIDEN_PLUS_EXPR
|| code == WIDEN_MINUS_EXPR
|| code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
+ || code == WIDEN_LSHIFT_EXPR
+ || code == IFN_VEC_WIDEN_PLUS_HILO
+ || code == IFN_VEC_WIDEN_MINUS_HILO);
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -5088,7 +5090,9 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
+ || code == WIDEN_MINUS_EXPR
+ || code == IFN_VEC_WIDEN_PLUS_HILO
+ || code == IFN_VEC_WIDEN_MINUS_HILO);
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
@@ -12478,10 +12482,43 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
+
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn ((combined_fn) code);
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ internal_fn lo, hi;
+ lookup_hilo_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = direct_internal_fn_optab (lo, {vectype, vectype});
+ optab2 = direct_internal_fn_optab (hi, {vectype, vectype});
+ }
+ else if (code.is_tree_code ())
{
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P ((tree_code) code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1, vectype, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ }
}
if (!optab1 || !optab2)
diff --git a/gcc/tree.def b/gcc/tree.def
index 90ceeec0b512bfa5f983359c0af03cc71de32007..b37b0b35927b92a6536e5c2d9805ffce8319a240 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1374,15 +1374,16 @@ DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr", tcc_expression, 3)
DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr", tcc_binary, 2)
/* Widening sad (sum of absolute differences).
- The first two arguments are of type t1 which should be integer.
- The third argument and the result are of type t2, such that t2 is at least
- twice the size of t1. Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
+ The first two arguments are of type t1 which should be a vector of integers.
+ The third argument and the result are of type t2, such that the size of
+ the elements of t2 is at least twice the size of the elements of t1.
+ Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
equivalent to:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = PLUS_EXPR (tmp2, arg3)
or:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = WIDEN_SUM_EXPR (tmp2, arg3)
*/
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 3/3] Remove widen_plus/minus_expr tree codes
2023-05-10 9:15 ` Andre Vieira (lists)
@ 2023-05-12 12:18 ` Andre Vieira (lists)
0 siblings, 0 replies; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-12 12:18 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, Richard Sandiford, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1583 bytes --]
Moved the 'changes' from this patch back to the second so it's all just
about removing code that we no longer use. I don't really know why Joel
formatted the patches this way, but I thought I'd keep it as is for now.
cover letter:
This patch removes the old widen plus/minus tree codes which have been
replaced by internal functions.
gcc/ChangeLog:
2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
* cfgexpand.cc (expand_debug_expr): Remove old tree codes.
* doc/generic.texi: Likewise.
* expr.cc (expand_expr_real_2): Likewise.
* gimple-pretty-print.cc (dump_binary_rhs): Likewise.
* gimple-range-op.cc (gimple_range_op_handler::maybe_non_standard):
Likewise.
* optabs-tree.cc (optab_for_tree_code): Likewise.
(supportable_half_widening_operation): Likewise.
* optabs.cc (commutative_optab_p): Likewise.
* optabs.def (OPTAB_D): Likewise.
* tree-cfg.cc (verify_gimple_assign_binary): Likewise.
* tree-inline.cc (estimate_operator_cost): Likewise.
(op_symbol_code): Likewise.
* tree-pretty-print.cc (dump_generic_node): Remove tree code definition.
* tree-vect-data-refs.cc (vect_get_smallest_scalar_type): Likewise.
(vect_analyze_data_ref_accesses): Likewise.
* tree-vect-generic.cc (expand_vector_operations_1): Likewise.
* tree-vect-stmts.cc (vectorizable_conversion): Likewise.
(supportable_widening_operation): Likewise.
* tree.def (WIDEN_PLUS_EXPR, WIDEN_MINUS_EXPR, VEC_WIDEN_PLUS_HI_EXPR,
VEC_WIDEN_PLUS_LO_EXPR, VEC_WIDEN_MINUS_HI_EXPR,
VEC_WIDEN_MINUS_LO_EXPR): Likewise.
[-- Attachment #2: ifn2v3.patch --]
[-- Type: text/plain, Size: 17304 bytes --]
diff --git a/gcc/cfgexpand.cc b/gcc/cfgexpand.cc
index 1a1b26b1c6c23ce273bcd08dc9a973f777174007..25b1558dcb941ea491a19aeeb2cd8f4d2dbdf7c6 100644
--- a/gcc/cfgexpand.cc
+++ b/gcc/cfgexpand.cc
@@ -5365,10 +5365,6 @@ expand_debug_expr (tree exp)
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_PERM_EXPR:
case VEC_DUPLICATE_EXPR:
case VEC_SERIES_EXPR:
@@ -5405,8 +5401,6 @@ expand_debug_expr (tree exp)
case WIDEN_MULT_EXPR:
case WIDEN_MULT_PLUS_EXPR:
case WIDEN_MULT_MINUS_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
if (SCALAR_INT_MODE_P (GET_MODE (op0))
&& SCALAR_INT_MODE_P (mode))
{
@@ -5419,10 +5413,6 @@ expand_debug_expr (tree exp)
op1 = simplify_gen_unary (ZERO_EXTEND, mode, op1, inner_mode);
else
op1 = simplify_gen_unary (SIGN_EXTEND, mode, op1, inner_mode);
- if (TREE_CODE (exp) == WIDEN_PLUS_EXPR)
- return simplify_gen_binary (PLUS, mode, op0, op1);
- else if (TREE_CODE (exp) == WIDEN_MINUS_EXPR)
- return simplify_gen_binary (MINUS, mode, op0, op1);
op0 = simplify_gen_binary (MULT, mode, op0, op1);
if (TREE_CODE (exp) == WIDEN_MULT_EXPR)
return op0;
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index 0fd7e6cce8bbd4ecb8027b702722adcf6c32eb55..a23d57af20610e0bb4809f06fb0c91253ae56d11 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1815,10 +1815,6 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex IFN_VEC_WIDEN_PLUS_LO
@tindex IFN_VEC_WIDEN_MINUS_HI
@tindex IFN_VEC_WIDEN_MINUS_LO
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1892,33 +1888,6 @@ vector of @code{N/2} products. In the case of
vector are subtracted from the low @code{N/2} of the first to produce the
vector of @code{N/2} products.
-@item VEC_WIDEN_PLUS_HI_EXPR
-@itemx VEC_WIDEN_PLUS_LO_EXPR
-These nodes represent widening vector addition of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The result
-is a vector that contains half as many elements, of an integral type whose size
-is twice as wide. In the case of @code{VEC_WIDEN_PLUS_HI_EXPR} the high
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products. In the case of @code{VEC_WIDEN_PLUS_LO_EXPR} the low
-@code{N/2} elements of the two vectors are added to produce the vector of
-@code{N/2} products.
-
-@item VEC_WIDEN_MINUS_HI_EXPR
-@itemx VEC_WIDEN_MINUS_LO_EXPR
-These nodes represent widening vector subtraction of the high and low parts of
-the two input vectors, respectively. Their operands are vectors that contain
-the same number of elements (@code{N}) of the same integral type. The high/low
-elements of the second vector are subtracted from the high/low elements of the
-first. The result is a vector that contains half as many elements, of an
-integral type whose size is twice as wide. In the case of
-@code{VEC_WIDEN_MINUS_HI_EXPR} the high @code{N/2} elements of the second
-vector are subtracted from the high @code{N/2} of the first to produce the
-vector of @code{N/2} products. In the case of
-@code{VEC_WIDEN_MINUS_LO_EXPR} the low @code{N/2} elements of the second
-vector are subtracted from the low @code{N/2} of the first to produce the
-vector of @code{N/2} products.
-
@item VEC_UNPACK_HI_EXPR
@itemx VEC_UNPACK_LO_EXPR
These nodes represent unpacking of the high and low parts of the input vector,
diff --git a/gcc/expr.cc b/gcc/expr.cc
index 758dda9ec68a8ba7a7b0e247aee50fd7996aa1d7..dd03688167b04299be213a5c379876499cb6a317 100644
--- a/gcc/expr.cc
+++ b/gcc/expr.cc
@@ -9600,8 +9600,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
target, unsignedp);
return target;
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_MULT_EXPR:
/* If first operand is constant, swap them.
Thus the following special case checks need only
@@ -10379,10 +10377,6 @@ expand_expr_real_2 (sepops ops, rtx target, machine_mode tmode,
return temp;
}
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/gimple-pretty-print.cc b/gcc/gimple-pretty-print.cc
index e46f7d5f55a31bf6453cd33683aa536f7fbe606f..8db221f65fe7e2fc1ce25685240f11516af87fe6 100644
--- a/gcc/gimple-pretty-print.cc
+++ b/gcc/gimple-pretty-print.cc
@@ -459,10 +459,6 @@ dump_binary_rhs (pretty_printer *buffer, const gassign *gs, int spc,
case VEC_PACK_FLOAT_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
case VEC_WIDEN_LSHIFT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_SERIES_EXPR:
for (p = get_tree_code_name (code); *p; p++)
pp_character (buffer, TOUPPER (*p));
diff --git a/gcc/gimple-range-op.cc b/gcc/gimple-range-op.cc
index 66636d82df27626e7911efd0cb8526921b39633f..466985bfd39a147d47ac525b7fe9bc3fd2d0b7b3 100644
--- a/gcc/gimple-range-op.cc
+++ b/gcc/gimple-range-op.cc
@@ -1191,12 +1191,6 @@ gimple_range_op_handler::maybe_non_standard ()
if (gimple_code (m_stmt) == GIMPLE_ASSIGN)
switch (gimple_assign_rhs_code (m_stmt))
{
- case WIDEN_PLUS_EXPR:
- {
- signed_op = ptr_op_widen_plus_signed;
- unsigned_op = ptr_op_widen_plus_unsigned;
- }
- gcc_fallthrough ();
case WIDEN_MULT_EXPR:
{
m_valid = false;
diff --git a/gcc/optabs-tree.cc b/gcc/optabs-tree.cc
index 8010046c6a8b3e809c989ddef7a06ddaa68ae32a..ee1aa8c9676ee9c67edbf403e6295da391826a62 100644
--- a/gcc/optabs-tree.cc
+++ b/gcc/optabs-tree.cc
@@ -190,22 +190,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
return (TYPE_UNSIGNED (type)
? vec_widen_ushiftl_lo_optab : vec_widen_sshiftl_lo_optab);
- case VEC_WIDEN_PLUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_lo_optab : vec_widen_saddl_lo_optab);
-
- case VEC_WIDEN_PLUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_uaddl_hi_optab : vec_widen_saddl_hi_optab);
-
- case VEC_WIDEN_MINUS_LO_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_lo_optab : vec_widen_ssubl_lo_optab);
-
- case VEC_WIDEN_MINUS_HI_EXPR:
- return (TYPE_UNSIGNED (type)
- ? vec_widen_usubl_hi_optab : vec_widen_ssubl_hi_optab);
-
case VEC_UNPACK_HI_EXPR:
return (TYPE_UNSIGNED (type)
? vec_unpacku_hi_optab : vec_unpacks_hi_optab);
@@ -312,8 +296,6 @@ optab_for_tree_code (enum tree_code code, const_tree type,
'hi'/'lo' pair using codes such as VEC_WIDEN_MINUS_HI/LO.
Supported widening operations:
- WIDEN_MINUS_EXPR
- WIDEN_PLUS_EXPR
WIDEN_MULT_EXPR
WIDEN_LSHIFT_EXPR
@@ -345,12 +327,6 @@ supportable_half_widening_operation (enum tree_code code, tree vectype_out,
case WIDEN_LSHIFT_EXPR:
*code1 = LSHIFT_EXPR;
break;
- case WIDEN_MINUS_EXPR:
- *code1 = MINUS_EXPR;
- break;
- case WIDEN_PLUS_EXPR:
- *code1 = PLUS_EXPR;
- break;
case WIDEN_MULT_EXPR:
*code1 = MULT_EXPR;
break;
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index 5a08d91e550b2d92e9572211f811fdba99a33a38..4309733a39be3d2a82dd2b13a50d73e6ddc2e0ff 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1315,10 +1315,6 @@ commutative_optab_p (optab binoptab)
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
|| binoptab == umul_highpart_optab
- || binoptab == vec_widen_saddl_hi_optab
- || binoptab == vec_widen_saddl_lo_optab
- || binoptab == vec_widen_uaddl_hi_optab
- || binoptab == vec_widen_uaddl_lo_optab
|| binoptab == vec_widen_sadd_hi_optab
|| binoptab == vec_widen_sadd_lo_optab
|| binoptab == vec_widen_uadd_hi_optab
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 16d121722c8c5723d9b164f5a2c616dc7ec143de..d4b3befdb822b98f12a9a440261f1b8e81432639 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -406,10 +406,6 @@ OPTAB_D (vec_widen_smult_even_optab, "vec_widen_smult_even_$a")
OPTAB_D (vec_widen_smult_hi_optab, "vec_widen_smult_hi_$a")
OPTAB_D (vec_widen_smult_lo_optab, "vec_widen_smult_lo_$a")
OPTAB_D (vec_widen_smult_odd_optab, "vec_widen_smult_odd_$a")
-OPTAB_D (vec_widen_ssubl_hi_optab, "vec_widen_ssubl_hi_$a")
-OPTAB_D (vec_widen_ssubl_lo_optab, "vec_widen_ssubl_lo_$a")
-OPTAB_D (vec_widen_saddl_hi_optab, "vec_widen_saddl_hi_$a")
-OPTAB_D (vec_widen_saddl_lo_optab, "vec_widen_saddl_lo_$a")
OPTAB_D (vec_widen_ssub_hi_optab, "vec_widen_ssub_hi_$a")
OPTAB_D (vec_widen_ssub_lo_optab, "vec_widen_ssub_lo_$a")
OPTAB_D (vec_widen_sadd_hi_optab, "vec_widen_sadd_hi_$a")
@@ -422,10 +418,6 @@ OPTAB_D (vec_widen_umult_lo_optab, "vec_widen_umult_lo_$a")
OPTAB_D (vec_widen_umult_odd_optab, "vec_widen_umult_odd_$a")
OPTAB_D (vec_widen_ushiftl_hi_optab, "vec_widen_ushiftl_hi_$a")
OPTAB_D (vec_widen_ushiftl_lo_optab, "vec_widen_ushiftl_lo_$a")
-OPTAB_D (vec_widen_usubl_hi_optab, "vec_widen_usubl_hi_$a")
-OPTAB_D (vec_widen_usubl_lo_optab, "vec_widen_usubl_lo_$a")
-OPTAB_D (vec_widen_uaddl_hi_optab, "vec_widen_uaddl_hi_$a")
-OPTAB_D (vec_widen_uaddl_lo_optab, "vec_widen_uaddl_lo_$a")
OPTAB_D (vec_widen_usub_hi_optab, "vec_widen_usub_hi_$a")
OPTAB_D (vec_widen_usub_lo_optab, "vec_widen_usub_lo_$a")
OPTAB_D (vec_widen_uadd_hi_optab, "vec_widen_uadd_hi_$a")
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index 28464ad9e3a7ea25557ffebcdbdbc1340f9e0d8b..dc28f5bbfa6272a92b68489fe67446bd3eba0caf 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -4068,8 +4068,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case PLUS_EXPR:
case MINUS_EXPR:
{
@@ -4190,10 +4188,6 @@ verify_gimple_assign_binary (gassign *stmt)
return false;
}
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index d74d8db2173b1ab117250fea89de5212d5e354ec..7b056c7dc7e173b0bc9981a5c98f0d50685b6b66 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4273,8 +4273,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case REALIGN_LOAD_EXPR:
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case DOT_PROD_EXPR:
@@ -4283,10 +4281,6 @@ estimate_operator_cost (enum tree_code code, eni_weights *weights,
case WIDEN_MULT_MINUS_EXPR:
case WIDEN_LSHIFT_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
diff --git a/gcc/tree-pretty-print.cc b/gcc/tree-pretty-print.cc
index 7947f9647a15110b52d195643ad7d28ee32d4236..9941d8bf80535a98e647b8928619a6bf08bc434c 100644
--- a/gcc/tree-pretty-print.cc
+++ b/gcc/tree-pretty-print.cc
@@ -2874,8 +2874,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
break;
/* Binary arithmetic and logic expressions. */
- case WIDEN_PLUS_EXPR:
- case WIDEN_MINUS_EXPR:
case WIDEN_SUM_EXPR:
case WIDEN_MULT_EXPR:
case MULT_EXPR:
@@ -3831,10 +3829,6 @@ dump_generic_node (pretty_printer *pp, tree node, int spc, dump_flags_t flags,
case VEC_SERIES_EXPR:
case VEC_WIDEN_MULT_HI_EXPR:
case VEC_WIDEN_MULT_LO_EXPR:
- case VEC_WIDEN_PLUS_HI_EXPR:
- case VEC_WIDEN_PLUS_LO_EXPR:
- case VEC_WIDEN_MINUS_HI_EXPR:
- case VEC_WIDEN_MINUS_LO_EXPR:
case VEC_WIDEN_MULT_EVEN_EXPR:
case VEC_WIDEN_MULT_ODD_EXPR:
case VEC_WIDEN_LSHIFT_HI_EXPR:
@@ -4352,12 +4346,6 @@ op_symbol_code (enum tree_code code)
case WIDEN_LSHIFT_EXPR:
return "w<<";
- case WIDEN_PLUS_EXPR:
- return "w+";
-
- case WIDEN_MINUS_EXPR:
- return "w-";
-
case POINTER_PLUS_EXPR:
return "+";
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 6721ab6efc4f029be8e2315c31ba87d94230cda5..68b29ee4661a0d08cf8f7048c23f992d4440b08b 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -136,8 +136,6 @@ vect_get_smallest_scalar_type (stmt_vec_info stmt_info, tree scalar_type)
|| gimple_assign_rhs_code (assign) == WIDEN_SUM_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_MULT_EXPR
|| gimple_assign_rhs_code (assign) == WIDEN_LSHIFT_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_PLUS_EXPR
- || gimple_assign_rhs_code (assign) == WIDEN_MINUS_EXPR
|| gimple_assign_rhs_code (assign) == FLOAT_EXPR)
{
tree rhs_type = TREE_TYPE (gimple_assign_rhs1 (assign));
diff --git a/gcc/tree-vect-generic.cc b/gcc/tree-vect-generic.cc
index 59115b2e1629358e85cb770f6da04cc5a2adb27a..7f55966310cee67238b2561e333ea45ee6153d9a 100644
--- a/gcc/tree-vect-generic.cc
+++ b/gcc/tree-vect-generic.cc
@@ -2198,10 +2198,6 @@ expand_vector_operations_1 (gimple_stmt_iterator *gsi,
arguments, not the widened result. VEC_UNPACK_FLOAT_*_EXPR is
calculated in the same way above. */
if (code == WIDEN_SUM_EXPR
- || code == VEC_WIDEN_PLUS_HI_EXPR
- || code == VEC_WIDEN_PLUS_LO_EXPR
- || code == VEC_WIDEN_MINUS_HI_EXPR
- || code == VEC_WIDEN_MINUS_LO_EXPR
|| code == VEC_WIDEN_MULT_HI_EXPR
|| code == VEC_WIDEN_MULT_LO_EXPR
|| code == VEC_WIDEN_MULT_EVEN_EXPR
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 24c811ebe01fb8b003100dea494cf64fea72a975..7a818a3b7ad4c9e6b1f45abcc1e4fbd056aa1d29 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5035,9 +5035,7 @@ vectorizable_conversion (vec_info *vinfo,
else
return false;
- bool widen_arith = (code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
- || code == WIDEN_MULT_EXPR
+ bool widen_arith = (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == IFN_VEC_WIDEN_PLUS_HILO
|| code == IFN_VEC_WIDEN_MINUS_HILO);
@@ -5089,8 +5087,6 @@ vectorizable_conversion (vec_info *vinfo,
{
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
- || code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR
|| code == IFN_VEC_WIDEN_PLUS_HILO
|| code == IFN_VEC_WIDEN_MINUS_HILO);
@@ -12335,7 +12331,7 @@ supportable_widening_operation (vec_info *vinfo,
class loop *vect_loop = NULL;
machine_mode vec_mode;
enum insn_code icode1, icode2;
- optab optab1, optab2;
+ optab optab1 = unknown_optab, optab2 = unknown_optab;
tree vectype = vectype_in;
tree wide_vectype = vectype_out;
tree_code c1 = MAX_TREE_CODES, c2 = MAX_TREE_CODES;
@@ -12433,16 +12429,6 @@ supportable_widening_operation (vec_info *vinfo,
c2 = VEC_WIDEN_LSHIFT_HI_EXPR;
break;
- case WIDEN_PLUS_EXPR:
- c1 = VEC_WIDEN_PLUS_LO_EXPR;
- c2 = VEC_WIDEN_PLUS_HI_EXPR;
- break;
-
- case WIDEN_MINUS_EXPR:
- c1 = VEC_WIDEN_MINUS_LO_EXPR;
- c2 = VEC_WIDEN_MINUS_HI_EXPR;
- break;
-
CASE_CONVERT:
c1 = VEC_UNPACK_LO_EXPR;
c2 = VEC_UNPACK_HI_EXPR;
diff --git a/gcc/tree.def b/gcc/tree.def
index b37b0b35927b92a6536e5c2d9805ffce8319a240..1fc2ca7a7249d4767aa2448219bc21a8c650aeb4 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1422,8 +1422,6 @@ DEFTREECODE (WIDEN_MULT_MINUS_EXPR, "widen_mult_minus_expr", tcc_expression, 3)
the first argument from type t1 to type t2, and then shifting it
by the second argument. */
DEFTREECODE (WIDEN_LSHIFT_EXPR, "widen_lshift_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_PLUS_EXPR, "widen_plus_expr", tcc_binary, 2)
-DEFTREECODE (WIDEN_MINUS_EXPR, "widen_minus_expr", tcc_binary, 2)
/* Widening vector multiplication.
The two operands are vectors with N elements of size S. Multiplying the
@@ -1488,10 +1486,6 @@ DEFTREECODE (VEC_PACK_FLOAT_EXPR, "vec_pack_float_expr", tcc_binary, 2)
*/
DEFTREECODE (VEC_WIDEN_LSHIFT_HI_EXPR, "widen_lshift_hi_expr", tcc_binary, 2)
DEFTREECODE (VEC_WIDEN_LSHIFT_LO_EXPR, "widen_lshift_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_HI_EXPR, "widen_plus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_PLUS_LO_EXPR, "widen_plus_lo_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_HI_EXPR, "widen_minus_hi_expr", tcc_binary, 2)
-DEFTREECODE (VEC_WIDEN_MINUS_LO_EXPR, "widen_minus_lo_expr", tcc_binary, 2)
/* PREDICT_EXPR. Specify hint for branch prediction. The
PREDICT_EXPR_PREDICTOR specify predictor and PREDICT_EXPR_OUTCOME the
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 1/3] Refactor to allow internal_fn's
2023-05-12 12:14 ` Andre Vieira (lists)
@ 2023-05-12 13:18 ` Richard Biener
0 siblings, 0 replies; 53+ messages in thread
From: Richard Biener @ 2023-05-12 13:18 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, Richard Sandiford, gcc-patches
On Fri, 12 May 2023, Andre Vieira (lists) wrote:
> Hi,
>
> I think I tackled all of your comments, let me know if I missed something.
This first and the last patch look good to me now. Let me comment on the
second.
Thanks,
Richard.
>
> gcc/ChangeLog:
>
> 2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
> Joel Hutton <joel.hutton@arm.com>
>
> * tree-vect-patterns.cc (vect_gimple_build): New Function.
> (vect_recog_widen_op_pattern): Refactor to use code_helper.
> * tree-vect-stmts.cc (vect_gen_widened_results_half): Likewise.
> (vect_create_vectorized_demotion_stmts): Likewise.
> (vect_create_vectorized_promotion_stmts): Likewise.
> (vect_create_half_widening_stmts): Likewise.
> (vectorizable_conversion): Likewise.
> (vectorizable_call): Likewise.
> (supportable_widening_operation): Likewise.
> (supportable_narrowing_operation): Likewise.
> (simple_integer_narrowing): Likewise.
> * tree-vectorizer.h (supportable_widening_operation): Likewise.
> (supportable_narrowing_operation): Likewise.
> (vect_gimple_build): New function prototype.
> * tree.h (code_helper::safe_as_tree_code): New function.
> (code_helper::safe_as_fn_code): New function.
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-12 12:16 ` Andre Vieira (lists)
@ 2023-05-12 13:28 ` Richard Biener
2023-05-12 13:55 ` Andre Vieira (lists)
2023-05-12 14:01 ` Richard Sandiford
0 siblings, 2 replies; 53+ messages in thread
From: Richard Biener @ 2023-05-12 13:28 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, gcc-patches, richard.sandiford
On Fri, 12 May 2023, Andre Vieira (lists) wrote:
> I have dealt with, I think..., most of your comments. There's quite a few
> changes, I think it's all a bit simpler now. I made some other changes to the
> costing in tree-inline.cc and gimple-range-op.cc in which I try to preserve
> the same behaviour as we had with the tree codes before. Also added some extra
> checks to tree-cfg.cc that made sense to me.
>
> I am still regression testing the gimple-range-op change, as that was a last
> minute change, but the rest survived a bootstrap and regression test on
> aarch64-unknown-linux-gnu.
>
> cover letter:
>
> This patch replaces the existing tree_code widen_plus and widen_minus
> patterns with internal_fn versions.
>
> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
> except they provide convenience wrappers for defining conversions that require
> a hi/lo split. Each definition for <NAME> will require optabs for _hi and _lo
> and each of those will also require a signed and unsigned version in the case
> of widening. The hi/lo pair is necessary because the widening and narrowing
> operations take n narrow elements as inputs and return n/2 wide elements as
> outputs. The 'lo' operation operates on the first n/2 elements of input. The
> 'hi' operation operates on the second n/2 elements of input. Defining an
> internal_fn along with hi/lo variations allows a single internal function to
> be returned from a vect_recog function that will later be expanded to hi/lo.
>
>
> For example:
> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
> (u/s)addl2
> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode>
> -> (u/s)addl
>
> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
What I still don't understand is how we are so narrowly focused on
HI/LO? We need a combined scalar IFN for pattern selection (not
sure why that's now called _HILO, I expected no suffix). Then there's
three possibilities the target can implement this:
1) with a widen_[su]add<mode> instruction - I _think_ that's what
RISCV is going to offer since it is a target where vector modes
have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
RVV can do a V4HI to V4SI widening and widening add/subtract
using vwadd[u] and vwsub[u] (the HI->SI widening is actually
done with a widening add of zero - eh).
IIRC GCN is the same here.
2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
codes currently support (exclusively)
3) similar, but widen_[su]add{_even,_odd}<mode>
that said, things like decomposes_to_hilo_fn_p look to paint us into
a 2) corner without good reason.
Richard.
> gcc/ChangeLog:
>
> 2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
> Joel Hutton <joel.hutton@arm.com>
> Tamar Christina <tamar.christina@arm.com>
>
> * config/aarch64/aarch64-simd.md (vec_widen_<su>addl_lo_<mode>):
> Rename
> this ...
> (vec_widen_<su>add_lo_<mode>): ... to this.
> (vec_widen_<su>addl_hi_<mode>): Rename this ...
> (vec_widen_<su>add_hi_<mode>): ... to this.
> (vec_widen_<su>subl_lo_<mode>): Rename this ...
> (vec_widen_<su>sub_lo_<mode>): ... to this.
> (vec_widen_<su>subl_hi_<mode>): Rename this ...
> (vec_widen_<su>sub_hi_<mode>): ...to this.
> * doc/generic.texi: Document new IFN codes.
> * internal-fn.cc (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Macro to
> define an
> internal_fn that expands into multiple internal_fns for widening.
> (DEF_INTERNAL_OPTAB_NARROWING_HILO_FN): Likewise but for narrowing.
> (ifn_cmp): Function to compare ifn's for sorting/searching.
> (lookup_hilo_internal_fn): Add lookup function.
> (commutative_binary_fn_p): Add widen_plus fn's.
> (widening_fn_p): New function.
> (narrowing_fn_p): New function.
> (decomposes_to_hilo_fn_p): New function.
> (direct_internal_fn_optab): Change visibility.
> * internal-fn.def (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Define
> widening
> plus,minus functions.
> (VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS_EXPR tree code.
> (VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS_EXPR tree code.
> * internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
> (direct_internal_fn_optab): Declare new prototype.
> (lookup_hilo_internal_fn): Likewise.
> (widening_fn_p): Likewise.
> (Narrowing_fn_p): Likewise.
> (decomposes_to_hilo_fn_p): Likewise.
> * optabs.cc (commutative_optab_p): Add widening plus optabs.
> * optabs.def (OPTAB_D): Define widen add, sub optabs.
> * tree-cfg.cc (verify_gimple_call): Add checks for new widen
> add and sub IFNs.
> * tree-inline.cc (estimate_num_insns): Return same
> cost for widen add and sub IFNs as previous tree_codes.
> * tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
> patterns with a hi/lo split.
> (vect_recog_sad_pattern): Refactor to use new IFN codes.
> (vect_recog_widen_plus_pattern): Likewise.
> (vect_recog_widen_minus_pattern): Likewise.
> (vect_recog_average_pattern): Likewise.
> * tree-vect-stmts.cc (vectorizable_conversion): Add support for
> _HILO IFNs.
> (supportable_widening_operation): Likewise.
> * tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
>
> gcc/testsuite/ChangeLog:
>
> * gcc.target/aarch64/vect-widen-add.c: Test that new
> IFN_VEC_WIDEN_PLUS is being used.
> * gcc.target/aarch64/vect-widen-sub.c: Test that new
> IFN_VEC_WIDEN_MINUS is being used.
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-12 13:28 ` Richard Biener
@ 2023-05-12 13:55 ` Andre Vieira (lists)
2023-05-12 14:01 ` Richard Sandiford
1 sibling, 0 replies; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-12 13:55 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Biener, gcc-patches, richard.sandiford
On 12/05/2023 14:28, Richard Biener wrote:
> On Fri, 12 May 2023, Andre Vieira (lists) wrote:
>
>> I have dealt with, I think..., most of your comments. There's quite a few
>> changes, I think it's all a bit simpler now. I made some other changes to the
>> costing in tree-inline.cc and gimple-range-op.cc in which I try to preserve
>> the same behaviour as we had with the tree codes before. Also added some extra
>> checks to tree-cfg.cc that made sense to me.
>>
>> I am still regression testing the gimple-range-op change, as that was a last
>> minute change, but the rest survived a bootstrap and regression test on
>> aarch64-unknown-linux-gnu.
>>
>> cover letter:
>>
>> This patch replaces the existing tree_code widen_plus and widen_minus
>> patterns with internal_fn versions.
>>
>> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
>> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
>> except they provide convenience wrappers for defining conversions that require
>> a hi/lo split. Each definition for <NAME> will require optabs for _hi and _lo
>> and each of those will also require a signed and unsigned version in the case
>> of widening. The hi/lo pair is necessary because the widening and narrowing
>> operations take n narrow elements as inputs and return n/2 wide elements as
>> outputs. The 'lo' operation operates on the first n/2 elements of input. The
>> 'hi' operation operates on the second n/2 elements of input. Defining an
>> internal_fn along with hi/lo variations allows a single internal function to
>> be returned from a vect_recog function that will later be expanded to hi/lo.
>>
>>
>> For example:
>> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
>> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
>> (u/s)addl2
>> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode>
>> -> (u/s)addl
>>
>> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
>> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
>
> What I still don't understand is how we are so narrowly focused on
> HI/LO? We need a combined scalar IFN for pattern selection (not
> sure why that's now called _HILO, I expected no suffix). Then there's
> three possibilities the target can implement this:
>
> 1) with a widen_[su]add<mode> instruction - I _think_ that's what
> RISCV is going to offer since it is a target where vector modes
> have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
> RVV can do a V4HI to V4SI widening and widening add/subtract
> using vwadd[u] and vwsub[u] (the HI->SI widening is actually
> done with a widening add of zero - eh).
> IIRC GCN is the same here.
> 2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
> codes currently support (exclusively)
> 3) similar, but widen_[su]add{_even,_odd}<mode>
>
> that said, things like decomposes_to_hilo_fn_p look to paint us into
> a 2) corner without good reason.
I was kind of just keeping the naming, I had forgotten to mention I was
also going to add _EVENODD but you are right, the pattern selection IFN
does not need to be restrictive.
And then at supportable_widening_operation we could check what the
target offers support for (either 1, 2 or 3). We can then actually just
get rid of decomposes_to_hilo_fn_p and just assume that for all
narrowing or widening IFN's there are optabs (that may or may not be
implemented by a target) for all three variants
Having said that, that means we should have an optab to cover 1, which
should probably just have the original name. Let me write it out...
Say we have a IFN_VEC_WIDEN_PLUS pattern and assume its signed,
supportable_widening_operation would then first check if the target
supported vec_widen_sadd_optab for say V8HI -> V8SI? Risc-V would take
this path I guess?
If the target doesn't then it could check for support for:
vec_widen_sadd_lo_optab V4HI -> V4SI
vec_widen_sadd_hi_optab V4HI -> V4SI
AArch64 Advanced SIMD would implement this.
If the target still didn't support this it would check for (not sure
about the modes here):
vec_widen_sadd_even_optab VNx8HI -> VNx4SI
vec_widen_sadd_odd_optab VNx8HI -> VNx4SI
This is one SVE would implement.
So that would mean that I'd probably end up rewriting
#define DEF_INTERNAL_OPTAB_WIDENING_FN (NAME, FLAGS, SELECTOR, SOPTAB,
UOPTAB, TYPE)
as:
for1)
DEF_INTERNAL_SIGNED_OPTAB_FN (NAME, FLAGS, SELECTOR, SOPTAB, UOPTAB,
TYPE)
for 2)
DEF_INTERNAL_SIGNED_OPTAB_FN (NAME##_LO, FLAGS, SELECTOR, SOPTAB,
UOPTAB, TYPE)
DEF_INTERNAL_SIGNED_OPTAB_FN (NAME##_HI, FLAGS, SELECTOR, SOPTAB,
UOPTAB, TYPE)
for 3)
DEF_INTERNAL_SIGNED_OPTAB_FN (NAME##_EVEN, FLAGS, SELECTOR, SOPTAB,
UOPTAB, TYPE)
DEF_INTERNAL_SIGNED_OPTAB_FN (NAME##_ODD, FLAGS, SELECTOR, SOPTAB,
UOPTAB, TYPE)
And the same for narrowing (but with DEF_INTERNAL_OPTAB_FN instead of
SIGNED_OPTAB).
So each widening and narrowing IFN would have optabs for all its
variants and each target implements the ones it supports.
I'm happy to do this, but implementing support to handle the 1 and 3
variants without having optabs for them right now seems a bit odd and it
would delay this patch, so I suggest I add the framework and the optabs
but leave adding the vectorizer support for later? I can add comments to
where I think that should go.
> Richard.
>
>> gcc/ChangeLog:
>>
>> 2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
>> Joel Hutton <joel.hutton@arm.com>
>> Tamar Christina <tamar.christina@arm.com>
>>
>> * config/aarch64/aarch64-simd.md (vec_widen_<su>addl_lo_<mode>):
>> Rename
>> this ...
>> (vec_widen_<su>add_lo_<mode>): ... to this.
>> (vec_widen_<su>addl_hi_<mode>): Rename this ...
>> (vec_widen_<su>add_hi_<mode>): ... to this.
>> (vec_widen_<su>subl_lo_<mode>): Rename this ...
>> (vec_widen_<su>sub_lo_<mode>): ... to this.
>> (vec_widen_<su>subl_hi_<mode>): Rename this ...
>> (vec_widen_<su>sub_hi_<mode>): ...to this.
>> * doc/generic.texi: Document new IFN codes.
>> * internal-fn.cc (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Macro to
>> define an
>> internal_fn that expands into multiple internal_fns for widening.
>> (DEF_INTERNAL_OPTAB_NARROWING_HILO_FN): Likewise but for narrowing.
>> (ifn_cmp): Function to compare ifn's for sorting/searching.
>> (lookup_hilo_internal_fn): Add lookup function.
>> (commutative_binary_fn_p): Add widen_plus fn's.
>> (widening_fn_p): New function.
>> (narrowing_fn_p): New function.
>> (decomposes_to_hilo_fn_p): New function.
>> (direct_internal_fn_optab): Change visibility.
>> * internal-fn.def (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Define
>> widening
>> plus,minus functions.
>> (VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS_EXPR tree code.
>> (VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS_EXPR tree code.
>> * internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
>> (direct_internal_fn_optab): Declare new prototype.
>> (lookup_hilo_internal_fn): Likewise.
>> (widening_fn_p): Likewise.
>> (Narrowing_fn_p): Likewise.
>> (decomposes_to_hilo_fn_p): Likewise.
>> * optabs.cc (commutative_optab_p): Add widening plus optabs.
>> * optabs.def (OPTAB_D): Define widen add, sub optabs.
>> * tree-cfg.cc (verify_gimple_call): Add checks for new widen
>> add and sub IFNs.
>> * tree-inline.cc (estimate_num_insns): Return same
>> cost for widen add and sub IFNs as previous tree_codes.
>> * tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
>> patterns with a hi/lo split.
>> (vect_recog_sad_pattern): Refactor to use new IFN codes.
>> (vect_recog_widen_plus_pattern): Likewise.
>> (vect_recog_widen_minus_pattern): Likewise.
>> (vect_recog_average_pattern): Likewise.
>> * tree-vect-stmts.cc (vectorizable_conversion): Add support for
>> _HILO IFNs.
>> (supportable_widening_operation): Likewise.
>> * tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
>>
>> gcc/testsuite/ChangeLog:
>>
>> * gcc.target/aarch64/vect-widen-add.c: Test that new
>> IFN_VEC_WIDEN_PLUS is being used.
>> * gcc.target/aarch64/vect-widen-sub.c: Test that new
>> IFN_VEC_WIDEN_MINUS is being used.
>>
>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-12 13:28 ` Richard Biener
2023-05-12 13:55 ` Andre Vieira (lists)
@ 2023-05-12 14:01 ` Richard Sandiford
2023-05-15 10:20 ` Richard Biener
1 sibling, 1 reply; 53+ messages in thread
From: Richard Sandiford @ 2023-05-12 14:01 UTC (permalink / raw)
To: Richard Biener; +Cc: Andre Vieira (lists), Richard Biener, gcc-patches
Richard Biener <rguenther@suse.de> writes:
> On Fri, 12 May 2023, Andre Vieira (lists) wrote:
>
>> I have dealt with, I think..., most of your comments. There's quite a few
>> changes, I think it's all a bit simpler now. I made some other changes to the
>> costing in tree-inline.cc and gimple-range-op.cc in which I try to preserve
>> the same behaviour as we had with the tree codes before. Also added some extra
>> checks to tree-cfg.cc that made sense to me.
>>
>> I am still regression testing the gimple-range-op change, as that was a last
>> minute change, but the rest survived a bootstrap and regression test on
>> aarch64-unknown-linux-gnu.
>>
>> cover letter:
>>
>> This patch replaces the existing tree_code widen_plus and widen_minus
>> patterns with internal_fn versions.
>>
>> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
>> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
>> except they provide convenience wrappers for defining conversions that require
>> a hi/lo split. Each definition for <NAME> will require optabs for _hi and _lo
>> and each of those will also require a signed and unsigned version in the case
>> of widening. The hi/lo pair is necessary because the widening and narrowing
>> operations take n narrow elements as inputs and return n/2 wide elements as
>> outputs. The 'lo' operation operates on the first n/2 elements of input. The
>> 'hi' operation operates on the second n/2 elements of input. Defining an
>> internal_fn along with hi/lo variations allows a single internal function to
>> be returned from a vect_recog function that will later be expanded to hi/lo.
>>
>>
>> For example:
>> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
>> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
>> (u/s)addl2
>> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode>
>> -> (u/s)addl
>>
>> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
>> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
>
> What I still don't understand is how we are so narrowly focused on
> HI/LO? We need a combined scalar IFN for pattern selection (not
> sure why that's now called _HILO, I expected no suffix). Then there's
> three possibilities the target can implement this:
>
> 1) with a widen_[su]add<mode> instruction - I _think_ that's what
> RISCV is going to offer since it is a target where vector modes
> have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
> RVV can do a V4HI to V4SI widening and widening add/subtract
> using vwadd[u] and vwsub[u] (the HI->SI widening is actually
> done with a widening add of zero - eh).
> IIRC GCN is the same here.
SVE currently does this too, but the addition and widening are
separate operations. E.g. in principle there's no reason why
you can't sign-extend one operand, zero-extend the other, and
then add the result together. Or you could extend them from
different sizes (QI and HI). All of those are supported
(if the costing allows them).
If the target has operations to do combined extending and adding (or
whatever), then at the moment we rely on combine to generate them.
So I think this case is separate from Andre's work. The addition
itself is just an ordinary addition, and any widening happens by
vectorising a CONVERT/NOP_EXPR.
> 2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
> codes currently support (exclusively)
> 3) similar, but widen_[su]add{_even,_odd}<mode>
>
> that said, things like decomposes_to_hilo_fn_p look to paint us into
> a 2) corner without good reason.
I suppose one question is: how much of the patch is really specific
to HI/LO, and how much is just grouping two halves together? The nice
thing about the internal-fn grouping macros is that, if (3) is
implemented in future, the structure will strongly encourage even/odd
pairs to be supported for all operations that support hi/lo. That is,
I would expect the grouping macros to be extended to define even/odd
ifns alongside hi/lo ones, rather than adding separate definitions
for even/odd functions.
If so, at least from the internal-fn.* side of things, I think the question
is whether it's OK to stick with hilo names for now, or whether we should
use more forward-looking names.
Thanks,
Richard
>
> Richard.
>
>> gcc/ChangeLog:
>>
>> 2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
>> Joel Hutton <joel.hutton@arm.com>
>> Tamar Christina <tamar.christina@arm.com>
>>
>> * config/aarch64/aarch64-simd.md (vec_widen_<su>addl_lo_<mode>):
>> Rename
>> this ...
>> (vec_widen_<su>add_lo_<mode>): ... to this.
>> (vec_widen_<su>addl_hi_<mode>): Rename this ...
>> (vec_widen_<su>add_hi_<mode>): ... to this.
>> (vec_widen_<su>subl_lo_<mode>): Rename this ...
>> (vec_widen_<su>sub_lo_<mode>): ... to this.
>> (vec_widen_<su>subl_hi_<mode>): Rename this ...
>> (vec_widen_<su>sub_hi_<mode>): ...to this.
>> * doc/generic.texi: Document new IFN codes.
>> * internal-fn.cc (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Macro to
>> define an
>> internal_fn that expands into multiple internal_fns for widening.
>> (DEF_INTERNAL_OPTAB_NARROWING_HILO_FN): Likewise but for narrowing.
>> (ifn_cmp): Function to compare ifn's for sorting/searching.
>> (lookup_hilo_internal_fn): Add lookup function.
>> (commutative_binary_fn_p): Add widen_plus fn's.
>> (widening_fn_p): New function.
>> (narrowing_fn_p): New function.
>> (decomposes_to_hilo_fn_p): New function.
>> (direct_internal_fn_optab): Change visibility.
>> * internal-fn.def (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Define
>> widening
>> plus,minus functions.
>> (VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS_EXPR tree code.
>> (VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS_EXPR tree code.
>> * internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
>> (direct_internal_fn_optab): Declare new prototype.
>> (lookup_hilo_internal_fn): Likewise.
>> (widening_fn_p): Likewise.
>> (Narrowing_fn_p): Likewise.
>> (decomposes_to_hilo_fn_p): Likewise.
>> * optabs.cc (commutative_optab_p): Add widening plus optabs.
>> * optabs.def (OPTAB_D): Define widen add, sub optabs.
>> * tree-cfg.cc (verify_gimple_call): Add checks for new widen
>> add and sub IFNs.
>> * tree-inline.cc (estimate_num_insns): Return same
>> cost for widen add and sub IFNs as previous tree_codes.
>> * tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
>> patterns with a hi/lo split.
>> (vect_recog_sad_pattern): Refactor to use new IFN codes.
>> (vect_recog_widen_plus_pattern): Likewise.
>> (vect_recog_widen_minus_pattern): Likewise.
>> (vect_recog_average_pattern): Likewise.
>> * tree-vect-stmts.cc (vectorizable_conversion): Add support for
>> _HILO IFNs.
>> (supportable_widening_operation): Likewise.
>> * tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
>>
>> gcc/testsuite/ChangeLog:
>>
>> * gcc.target/aarch64/vect-widen-add.c: Test that new
>> IFN_VEC_WIDEN_PLUS is being used.
>> * gcc.target/aarch64/vect-widen-sub.c: Test that new
>> IFN_VEC_WIDEN_MINUS is being used.
>>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-12 14:01 ` Richard Sandiford
@ 2023-05-15 10:20 ` Richard Biener
2023-05-15 10:47 ` Richard Sandiford
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-05-15 10:20 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Andre Vieira (lists), Richard Biener, gcc-patches
On Fri, 12 May 2023, Richard Sandiford wrote:
> Richard Biener <rguenther@suse.de> writes:
> > On Fri, 12 May 2023, Andre Vieira (lists) wrote:
> >
> >> I have dealt with, I think..., most of your comments. There's quite a few
> >> changes, I think it's all a bit simpler now. I made some other changes to the
> >> costing in tree-inline.cc and gimple-range-op.cc in which I try to preserve
> >> the same behaviour as we had with the tree codes before. Also added some extra
> >> checks to tree-cfg.cc that made sense to me.
> >>
> >> I am still regression testing the gimple-range-op change, as that was a last
> >> minute change, but the rest survived a bootstrap and regression test on
> >> aarch64-unknown-linux-gnu.
> >>
> >> cover letter:
> >>
> >> This patch replaces the existing tree_code widen_plus and widen_minus
> >> patterns with internal_fn versions.
> >>
> >> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
> >> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
> >> except they provide convenience wrappers for defining conversions that require
> >> a hi/lo split. Each definition for <NAME> will require optabs for _hi and _lo
> >> and each of those will also require a signed and unsigned version in the case
> >> of widening. The hi/lo pair is necessary because the widening and narrowing
> >> operations take n narrow elements as inputs and return n/2 wide elements as
> >> outputs. The 'lo' operation operates on the first n/2 elements of input. The
> >> 'hi' operation operates on the second n/2 elements of input. Defining an
> >> internal_fn along with hi/lo variations allows a single internal function to
> >> be returned from a vect_recog function that will later be expanded to hi/lo.
> >>
> >>
> >> For example:
> >> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
> >> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
> >> (u/s)addl2
> >> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode>
> >> -> (u/s)addl
> >>
> >> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
> >> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
> >
> > What I still don't understand is how we are so narrowly focused on
> > HI/LO? We need a combined scalar IFN for pattern selection (not
> > sure why that's now called _HILO, I expected no suffix). Then there's
> > three possibilities the target can implement this:
> >
> > 1) with a widen_[su]add<mode> instruction - I _think_ that's what
> > RISCV is going to offer since it is a target where vector modes
> > have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
> > RVV can do a V4HI to V4SI widening and widening add/subtract
> > using vwadd[u] and vwsub[u] (the HI->SI widening is actually
> > done with a widening add of zero - eh).
> > IIRC GCN is the same here.
>
> SVE currently does this too, but the addition and widening are
> separate operations. E.g. in principle there's no reason why
> you can't sign-extend one operand, zero-extend the other, and
> then add the result together. Or you could extend them from
> different sizes (QI and HI). All of those are supported
> (if the costing allows them).
I see. So why does the target the expose widen_[su]add<mode> at all?
> If the target has operations to do combined extending and adding (or
> whatever), then at the moment we rely on combine to generate them.
>
> So I think this case is separate from Andre's work. The addition
> itself is just an ordinary addition, and any widening happens by
> vectorising a CONVERT/NOP_EXPR.
>
> > 2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
> > codes currently support (exclusively)
> > 3) similar, but widen_[su]add{_even,_odd}<mode>
> >
> > that said, things like decomposes_to_hilo_fn_p look to paint us into
> > a 2) corner without good reason.
>
> I suppose one question is: how much of the patch is really specific
> to HI/LO, and how much is just grouping two halves together?
Yep, that I don't know for sure.
> The nice
> thing about the internal-fn grouping macros is that, if (3) is
> implemented in future, the structure will strongly encourage even/odd
> pairs to be supported for all operations that support hi/lo. That is,
> I would expect the grouping macros to be extended to define even/odd
> ifns alongside hi/lo ones, rather than adding separate definitions
> for even/odd functions.
>
> If so, at least from the internal-fn.* side of things, I think the question
> is whether it's OK to stick with hilo names for now, or whether we should
> use more forward-looking names.
I think for parts that are independent we could use a more
forward-looking name. Maybe _halves? But I'm also not sure
how much of that is really needed (it seems to be tied around
optimizing optabs space?)
Richard.
> Thanks,
> Richard
>
> >
> > Richard.
> >
> >> gcc/ChangeLog:
> >>
> >> 2023-05-12 Andre Vieira <andre.simoesdiasvieira@arm.com>
> >> Joel Hutton <joel.hutton@arm.com>
> >> Tamar Christina <tamar.christina@arm.com>
> >>
> >> * config/aarch64/aarch64-simd.md (vec_widen_<su>addl_lo_<mode>):
> >> Rename
> >> this ...
> >> (vec_widen_<su>add_lo_<mode>): ... to this.
> >> (vec_widen_<su>addl_hi_<mode>): Rename this ...
> >> (vec_widen_<su>add_hi_<mode>): ... to this.
> >> (vec_widen_<su>subl_lo_<mode>): Rename this ...
> >> (vec_widen_<su>sub_lo_<mode>): ... to this.
> >> (vec_widen_<su>subl_hi_<mode>): Rename this ...
> >> (vec_widen_<su>sub_hi_<mode>): ...to this.
> >> * doc/generic.texi: Document new IFN codes.
> >> * internal-fn.cc (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Macro to
> >> define an
> >> internal_fn that expands into multiple internal_fns for widening.
> >> (DEF_INTERNAL_OPTAB_NARROWING_HILO_FN): Likewise but for narrowing.
> >> (ifn_cmp): Function to compare ifn's for sorting/searching.
> >> (lookup_hilo_internal_fn): Add lookup function.
> >> (commutative_binary_fn_p): Add widen_plus fn's.
> >> (widening_fn_p): New function.
> >> (narrowing_fn_p): New function.
> >> (decomposes_to_hilo_fn_p): New function.
> >> (direct_internal_fn_optab): Change visibility.
> >> * internal-fn.def (DEF_INTERNAL_OPTAB_WIDENING_HILO_FN): Define
> >> widening
> >> plus,minus functions.
> >> (VEC_WIDEN_PLUS): Replacement for VEC_WIDEN_PLUS_EXPR tree code.
> >> (VEC_WIDEN_MINUS): Replacement for VEC_WIDEN_MINUS_EXPR tree code.
> >> * internal-fn.h (GCC_INTERNAL_FN_H): Add headers.
> >> (direct_internal_fn_optab): Declare new prototype.
> >> (lookup_hilo_internal_fn): Likewise.
> >> (widening_fn_p): Likewise.
> >> (Narrowing_fn_p): Likewise.
> >> (decomposes_to_hilo_fn_p): Likewise.
> >> * optabs.cc (commutative_optab_p): Add widening plus optabs.
> >> * optabs.def (OPTAB_D): Define widen add, sub optabs.
> >> * tree-cfg.cc (verify_gimple_call): Add checks for new widen
> >> add and sub IFNs.
> >> * tree-inline.cc (estimate_num_insns): Return same
> >> cost for widen add and sub IFNs as previous tree_codes.
> >> * tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
> >> patterns with a hi/lo split.
> >> (vect_recog_sad_pattern): Refactor to use new IFN codes.
> >> (vect_recog_widen_plus_pattern): Likewise.
> >> (vect_recog_widen_minus_pattern): Likewise.
> >> (vect_recog_average_pattern): Likewise.
> >> * tree-vect-stmts.cc (vectorizable_conversion): Add support for
> >> _HILO IFNs.
> >> (supportable_widening_operation): Likewise.
> >> * tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
> >>
> >> gcc/testsuite/ChangeLog:
> >>
> >> * gcc.target/aarch64/vect-widen-add.c: Test that new
> >> IFN_VEC_WIDEN_PLUS is being used.
> >> * gcc.target/aarch64/vect-widen-sub.c: Test that new
> >> IFN_VEC_WIDEN_MINUS is being used.
> >>
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-15 10:20 ` Richard Biener
@ 2023-05-15 10:47 ` Richard Sandiford
2023-05-15 11:01 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Richard Sandiford @ 2023-05-15 10:47 UTC (permalink / raw)
To: Richard Biener; +Cc: Andre Vieira (lists), Richard Biener, gcc-patches
Richard Biener <rguenther@suse.de> writes:
> On Fri, 12 May 2023, Richard Sandiford wrote:
>
>> Richard Biener <rguenther@suse.de> writes:
>> > On Fri, 12 May 2023, Andre Vieira (lists) wrote:
>> >
>> >> I have dealt with, I think..., most of your comments. There's quite a few
>> >> changes, I think it's all a bit simpler now. I made some other changes to the
>> >> costing in tree-inline.cc and gimple-range-op.cc in which I try to preserve
>> >> the same behaviour as we had with the tree codes before. Also added some extra
>> >> checks to tree-cfg.cc that made sense to me.
>> >>
>> >> I am still regression testing the gimple-range-op change, as that was a last
>> >> minute change, but the rest survived a bootstrap and regression test on
>> >> aarch64-unknown-linux-gnu.
>> >>
>> >> cover letter:
>> >>
>> >> This patch replaces the existing tree_code widen_plus and widen_minus
>> >> patterns with internal_fn versions.
>> >>
>> >> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
>> >> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
>> >> except they provide convenience wrappers for defining conversions that require
>> >> a hi/lo split. Each definition for <NAME> will require optabs for _hi and _lo
>> >> and each of those will also require a signed and unsigned version in the case
>> >> of widening. The hi/lo pair is necessary because the widening and narrowing
>> >> operations take n narrow elements as inputs and return n/2 wide elements as
>> >> outputs. The 'lo' operation operates on the first n/2 elements of input. The
>> >> 'hi' operation operates on the second n/2 elements of input. Defining an
>> >> internal_fn along with hi/lo variations allows a single internal function to
>> >> be returned from a vect_recog function that will later be expanded to hi/lo.
>> >>
>> >>
>> >> For example:
>> >> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
>> >> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
>> >> (u/s)addl2
>> >> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode>
>> >> -> (u/s)addl
>> >>
>> >> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
>> >> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
>> >
>> > What I still don't understand is how we are so narrowly focused on
>> > HI/LO? We need a combined scalar IFN for pattern selection (not
>> > sure why that's now called _HILO, I expected no suffix). Then there's
>> > three possibilities the target can implement this:
>> >
>> > 1) with a widen_[su]add<mode> instruction - I _think_ that's what
>> > RISCV is going to offer since it is a target where vector modes
>> > have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
>> > RVV can do a V4HI to V4SI widening and widening add/subtract
>> > using vwadd[u] and vwsub[u] (the HI->SI widening is actually
>> > done with a widening add of zero - eh).
>> > IIRC GCN is the same here.
>>
>> SVE currently does this too, but the addition and widening are
>> separate operations. E.g. in principle there's no reason why
>> you can't sign-extend one operand, zero-extend the other, and
>> then add the result together. Or you could extend them from
>> different sizes (QI and HI). All of those are supported
>> (if the costing allows them).
>
> I see. So why does the target the expose widen_[su]add<mode> at all?
It shouldn't (need to) do that. I don't think we should have an optab
for the unsplit operation.
At least on SVE, we really want the extensions to be fused with loads
(where possible) rather than with arithmetic.
We can still do the widening arithmetic in one go. It's just that
fusing with the loads works for the mixed-sign and mixed-size cases,
and can handle more than just doubling the element size.
>> If the target has operations to do combined extending and adding (or
>> whatever), then at the moment we rely on combine to generate them.
>>
>> So I think this case is separate from Andre's work. The addition
>> itself is just an ordinary addition, and any widening happens by
>> vectorising a CONVERT/NOP_EXPR.
>>
>> > 2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
>> > codes currently support (exclusively)
>> > 3) similar, but widen_[su]add{_even,_odd}<mode>
>> >
>> > that said, things like decomposes_to_hilo_fn_p look to paint us into
>> > a 2) corner without good reason.
>>
>> I suppose one question is: how much of the patch is really specific
>> to HI/LO, and how much is just grouping two halves together?
>
> Yep, that I don't know for sure.
>
>> The nice
>> thing about the internal-fn grouping macros is that, if (3) is
>> implemented in future, the structure will strongly encourage even/odd
>> pairs to be supported for all operations that support hi/lo. That is,
>> I would expect the grouping macros to be extended to define even/odd
>> ifns alongside hi/lo ones, rather than adding separate definitions
>> for even/odd functions.
>>
>> If so, at least from the internal-fn.* side of things, I think the question
>> is whether it's OK to stick with hilo names for now, or whether we should
>> use more forward-looking names.
>
> I think for parts that are independent we could use a more
> forward-looking name. Maybe _halves?
Using _halves for the ifn macros sounds good to me FWIW.
> But I'm also not sure
> how much of that is really needed (it seems to be tied around
> optimizing optabs space?)
Not sure what you mean by "this". Optabs space shouldn't be a problem
though. The optab encoding gives us a full int to play with, and it
could easily go up to 64 bits if necessary/convenient.
At least on the internal-fn.* side, the aim is really just to establish
a regular structure, so that we don't have arbitrary differences between
different widening operations, or too much cut-&-paste.
Thanks,
Richard
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-15 10:47 ` Richard Sandiford
@ 2023-05-15 11:01 ` Richard Biener
2023-05-15 11:10 ` Richard Sandiford
2023-05-15 11:53 ` Andre Vieira (lists)
0 siblings, 2 replies; 53+ messages in thread
From: Richard Biener @ 2023-05-15 11:01 UTC (permalink / raw)
To: Richard Sandiford; +Cc: Andre Vieira (lists), Richard Biener, gcc-patches
On Mon, 15 May 2023, Richard Sandiford wrote:
> Richard Biener <rguenther@suse.de> writes:
> > On Fri, 12 May 2023, Richard Sandiford wrote:
> >
> >> Richard Biener <rguenther@suse.de> writes:
> >> > On Fri, 12 May 2023, Andre Vieira (lists) wrote:
> >> >
> >> >> I have dealt with, I think..., most of your comments. There's quite a few
> >> >> changes, I think it's all a bit simpler now. I made some other changes to the
> >> >> costing in tree-inline.cc and gimple-range-op.cc in which I try to preserve
> >> >> the same behaviour as we had with the tree codes before. Also added some extra
> >> >> checks to tree-cfg.cc that made sense to me.
> >> >>
> >> >> I am still regression testing the gimple-range-op change, as that was a last
> >> >> minute change, but the rest survived a bootstrap and regression test on
> >> >> aarch64-unknown-linux-gnu.
> >> >>
> >> >> cover letter:
> >> >>
> >> >> This patch replaces the existing tree_code widen_plus and widen_minus
> >> >> patterns with internal_fn versions.
> >> >>
> >> >> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
> >> >> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
> >> >> except they provide convenience wrappers for defining conversions that require
> >> >> a hi/lo split. Each definition for <NAME> will require optabs for _hi and _lo
> >> >> and each of those will also require a signed and unsigned version in the case
> >> >> of widening. The hi/lo pair is necessary because the widening and narrowing
> >> >> operations take n narrow elements as inputs and return n/2 wide elements as
> >> >> outputs. The 'lo' operation operates on the first n/2 elements of input. The
> >> >> 'hi' operation operates on the second n/2 elements of input. Defining an
> >> >> internal_fn along with hi/lo variations allows a single internal function to
> >> >> be returned from a vect_recog function that will later be expanded to hi/lo.
> >> >>
> >> >>
> >> >> For example:
> >> >> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
> >> >> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
> >> >> (u/s)addl2
> >> >> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode>
> >> >> -> (u/s)addl
> >> >>
> >> >> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
> >> >> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
> >> >
> >> > What I still don't understand is how we are so narrowly focused on
> >> > HI/LO? We need a combined scalar IFN for pattern selection (not
> >> > sure why that's now called _HILO, I expected no suffix). Then there's
> >> > three possibilities the target can implement this:
> >> >
> >> > 1) with a widen_[su]add<mode> instruction - I _think_ that's what
> >> > RISCV is going to offer since it is a target where vector modes
> >> > have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
> >> > RVV can do a V4HI to V4SI widening and widening add/subtract
> >> > using vwadd[u] and vwsub[u] (the HI->SI widening is actually
> >> > done with a widening add of zero - eh).
> >> > IIRC GCN is the same here.
> >>
> >> SVE currently does this too, but the addition and widening are
> >> separate operations. E.g. in principle there's no reason why
> >> you can't sign-extend one operand, zero-extend the other, and
> >> then add the result together. Or you could extend them from
> >> different sizes (QI and HI). All of those are supported
> >> (if the costing allows them).
> >
> > I see. So why does the target the expose widen_[su]add<mode> at all?
>
> It shouldn't (need to) do that. I don't think we should have an optab
> for the unsplit operation.
>
> At least on SVE, we really want the extensions to be fused with loads
> (where possible) rather than with arithmetic.
>
> We can still do the widening arithmetic in one go. It's just that
> fusing with the loads works for the mixed-sign and mixed-size cases,
> and can handle more than just doubling the element size.
>
> >> If the target has operations to do combined extending and adding (or
> >> whatever), then at the moment we rely on combine to generate them.
> >>
> >> So I think this case is separate from Andre's work. The addition
> >> itself is just an ordinary addition, and any widening happens by
> >> vectorising a CONVERT/NOP_EXPR.
> >>
> >> > 2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
> >> > codes currently support (exclusively)
> >> > 3) similar, but widen_[su]add{_even,_odd}<mode>
> >> >
> >> > that said, things like decomposes_to_hilo_fn_p look to paint us into
> >> > a 2) corner without good reason.
> >>
> >> I suppose one question is: how much of the patch is really specific
> >> to HI/LO, and how much is just grouping two halves together?
> >
> > Yep, that I don't know for sure.
> >
> >> The nice
> >> thing about the internal-fn grouping macros is that, if (3) is
> >> implemented in future, the structure will strongly encourage even/odd
> >> pairs to be supported for all operations that support hi/lo. That is,
> >> I would expect the grouping macros to be extended to define even/odd
> >> ifns alongside hi/lo ones, rather than adding separate definitions
> >> for even/odd functions.
> >>
> >> If so, at least from the internal-fn.* side of things, I think the question
> >> is whether it's OK to stick with hilo names for now, or whether we should
> >> use more forward-looking names.
> >
> > I think for parts that are independent we could use a more
> > forward-looking name. Maybe _halves?
>
> Using _halves for the ifn macros sounds good to me FWIW.
>
> > But I'm also not sure
> > how much of that is really needed (it seems to be tied around
> > optimizing optabs space?)
>
> Not sure what you mean by "this". Optabs space shouldn't be a problem
> though. The optab encoding gives us a full int to play with, and it
> could easily go up to 64 bits if necessary/convenient.
>
> At least on the internal-fn.* side, the aim is really just to establish
> a regular structure, so that we don't have arbitrary differences between
> different widening operations, or too much cut-&-paste.
Hmm, I'm looking at the need for the std::map and
internal_fn_hilo_keys_array and internal_fn_hilo_values_array.
The vectorizer pieces contain
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn ((combined_fn) code);
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ internal_fn lo, hi;
+ lookup_hilo_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = lookup_hilo_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
+ optab2 = lookup_hilo_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
so that tries to automatically associate the scalar widening IFN
with the set(s) of IFN pairs we can split to. But then this
list should be static and there's no need to create a std::map?
Maybe gencfn-macros.cc can be enhanced to output these static
cases? Or the vectorizer could (as it did previously) simply
open-code the handled cases (I guess since we deal with two
cases only now I'd prefer that).
Thanks,
Richard.
> Thanks,
> Richard
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-15 11:01 ` Richard Biener
@ 2023-05-15 11:10 ` Richard Sandiford
2023-05-15 11:53 ` Andre Vieira (lists)
1 sibling, 0 replies; 53+ messages in thread
From: Richard Sandiford @ 2023-05-15 11:10 UTC (permalink / raw)
To: Richard Biener; +Cc: Andre Vieira (lists), Richard Biener, gcc-patches
Richard Biener <rguenther@suse.de> writes:
> On Mon, 15 May 2023, Richard Sandiford wrote:
>
>> Richard Biener <rguenther@suse.de> writes:
>> > But I'm also not sure
>> > how much of that is really needed (it seems to be tied around
>> > optimizing optabs space?)
>>
>> Not sure what you mean by "this". Optabs space shouldn't be a problem
>> though. The optab encoding gives us a full int to play with, and it
>> could easily go up to 64 bits if necessary/convenient.
>>
>> At least on the internal-fn.* side, the aim is really just to establish
>> a regular structure, so that we don't have arbitrary differences between
>> different widening operations, or too much cut-&-paste.
>
> Hmm, I'm looking at the need for the std::map and
> internal_fn_hilo_keys_array and internal_fn_hilo_values_array.
> The vectorizer pieces contain
>
> + if (code.is_fn_code ())
> + {
> + internal_fn ifn = as_internal_fn ((combined_fn) code);
> + gcc_assert (decomposes_to_hilo_fn_p (ifn));
> +
> + internal_fn lo, hi;
> + lookup_hilo_internal_fn (ifn, &lo, &hi);
> + *code1 = as_combined_fn (lo);
> + *code2 = as_combined_fn (hi);
> + optab1 = lookup_hilo_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
> + optab2 = lookup_hilo_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
>
> so that tries to automatically associate the scalar widening IFN
> with the set(s) of IFN pairs we can split to. But then this
> list should be static and there's no need to create a std::map?
> Maybe gencfn-macros.cc can be enhanced to output these static
> cases? Or the vectorizer could (as it did previously) simply
> open-code the handled cases (I guess since we deal with two
> cases only now I'd prefer that).
Ah, yeah, I pushed back against that too. I think it should be possible
to do it using the preprocessor, if the macros are defined appropriately.
But if it isn't possible to do it with macros then I agree that a
generator would be better than initialisation within the compiler.
Thanks,
Richard
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-15 11:01 ` Richard Biener
2023-05-15 11:10 ` Richard Sandiford
@ 2023-05-15 11:53 ` Andre Vieira (lists)
2023-05-15 12:21 ` Richard Biener
1 sibling, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-15 11:53 UTC (permalink / raw)
To: Richard Biener, Richard Sandiford; +Cc: Richard Biener, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 7818 bytes --]
On 15/05/2023 12:01, Richard Biener wrote:
> On Mon, 15 May 2023, Richard Sandiford wrote:
>
>> Richard Biener <rguenther@suse.de> writes:
>>> On Fri, 12 May 2023, Richard Sandiford wrote:
>>>
>>>> Richard Biener <rguenther@suse.de> writes:
>>>>> On Fri, 12 May 2023, Andre Vieira (lists) wrote:
>>>>>
>>>>>> I have dealt with, I think..., most of your comments. There's quite a few
>>>>>> changes, I think it's all a bit simpler now. I made some other changes to the
>>>>>> costing in tree-inline.cc and gimple-range-op.cc in which I try to preserve
>>>>>> the same behaviour as we had with the tree codes before. Also added some extra
>>>>>> checks to tree-cfg.cc that made sense to me.
>>>>>>
>>>>>> I am still regression testing the gimple-range-op change, as that was a last
>>>>>> minute change, but the rest survived a bootstrap and regression test on
>>>>>> aarch64-unknown-linux-gnu.
>>>>>>
>>>>>> cover letter:
>>>>>>
>>>>>> This patch replaces the existing tree_code widen_plus and widen_minus
>>>>>> patterns with internal_fn versions.
>>>>>>
>>>>>> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
>>>>>> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN respectively
>>>>>> except they provide convenience wrappers for defining conversions that require
>>>>>> a hi/lo split. Each definition for <NAME> will require optabs for _hi and _lo
>>>>>> and each of those will also require a signed and unsigned version in the case
>>>>>> of widening. The hi/lo pair is necessary because the widening and narrowing
>>>>>> operations take n narrow elements as inputs and return n/2 wide elements as
>>>>>> outputs. The 'lo' operation operates on the first n/2 elements of input. The
>>>>>> 'hi' operation operates on the second n/2 elements of input. Defining an
>>>>>> internal_fn along with hi/lo variations allows a single internal function to
>>>>>> be returned from a vect_recog function that will later be expanded to hi/lo.
>>>>>>
>>>>>>
>>>>>> For example:
>>>>>> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
>>>>>> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
>>>>>> (u/s)addl2
>>>>>> IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode>
>>>>>> -> (u/s)addl
>>>>>>
>>>>>> This gives the same functionality as the previous WIDEN_PLUS/WIDEN_MINUS tree
>>>>>> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
>>>>>
>>>>> What I still don't understand is how we are so narrowly focused on
>>>>> HI/LO? We need a combined scalar IFN for pattern selection (not
>>>>> sure why that's now called _HILO, I expected no suffix). Then there's
>>>>> three possibilities the target can implement this:
>>>>>
>>>>> 1) with a widen_[su]add<mode> instruction - I _think_ that's what
>>>>> RISCV is going to offer since it is a target where vector modes
>>>>> have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
>>>>> RVV can do a V4HI to V4SI widening and widening add/subtract
>>>>> using vwadd[u] and vwsub[u] (the HI->SI widening is actually
>>>>> done with a widening add of zero - eh).
>>>>> IIRC GCN is the same here.
>>>>
>>>> SVE currently does this too, but the addition and widening are
>>>> separate operations. E.g. in principle there's no reason why
>>>> you can't sign-extend one operand, zero-extend the other, and
>>>> then add the result together. Or you could extend them from
>>>> different sizes (QI and HI). All of those are supported
>>>> (if the costing allows them).
>>>
>>> I see. So why does the target the expose widen_[su]add<mode> at all?
>>
>> It shouldn't (need to) do that. I don't think we should have an optab
>> for the unsplit operation.
>>
>> At least on SVE, we really want the extensions to be fused with loads
>> (where possible) rather than with arithmetic.
>>
>> We can still do the widening arithmetic in one go. It's just that
>> fusing with the loads works for the mixed-sign and mixed-size cases,
>> and can handle more than just doubling the element size.
>>
>>>> If the target has operations to do combined extending and adding (or
>>>> whatever), then at the moment we rely on combine to generate them.
>>>>
>>>> So I think this case is separate from Andre's work. The addition
>>>> itself is just an ordinary addition, and any widening happens by
>>>> vectorising a CONVERT/NOP_EXPR.
>>>>
>>>>> 2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
>>>>> codes currently support (exclusively)
>>>>> 3) similar, but widen_[su]add{_even,_odd}<mode>
>>>>>
>>>>> that said, things like decomposes_to_hilo_fn_p look to paint us into
>>>>> a 2) corner without good reason.
>>>>
>>>> I suppose one question is: how much of the patch is really specific
>>>> to HI/LO, and how much is just grouping two halves together?
>>>
>>> Yep, that I don't know for sure.
>>>
>>>> The nice
>>>> thing about the internal-fn grouping macros is that, if (3) is
>>>> implemented in future, the structure will strongly encourage even/odd
>>>> pairs to be supported for all operations that support hi/lo. That is,
>>>> I would expect the grouping macros to be extended to define even/odd
>>>> ifns alongside hi/lo ones, rather than adding separate definitions
>>>> for even/odd functions.
>>>>
>>>> If so, at least from the internal-fn.* side of things, I think the question
>>>> is whether it's OK to stick with hilo names for now, or whether we should
>>>> use more forward-looking names.
>>>
>>> I think for parts that are independent we could use a more
>>> forward-looking name. Maybe _halves?
>>
>> Using _halves for the ifn macros sounds good to me FWIW.
>>
>>> But I'm also not sure
>>> how much of that is really needed (it seems to be tied around
>>> optimizing optabs space?)
>>
>> Not sure what you mean by "this". Optabs space shouldn't be a problem
>> though. The optab encoding gives us a full int to play with, and it
>> could easily go up to 64 bits if necessary/convenient.
>>
>> At least on the internal-fn.* side, the aim is really just to establish
>> a regular structure, so that we don't have arbitrary differences between
>> different widening operations, or too much cut-&-paste.
>
> Hmm, I'm looking at the need for the std::map and
> internal_fn_hilo_keys_array and internal_fn_hilo_values_array.
> The vectorizer pieces contain
>
> + if (code.is_fn_code ())
> + {
> + internal_fn ifn = as_internal_fn ((combined_fn) code);
> + gcc_assert (decomposes_to_hilo_fn_p (ifn));
> +
> + internal_fn lo, hi;
> + lookup_hilo_internal_fn (ifn, &lo, &hi);
> + *code1 = as_combined_fn (lo);
> + *code2 = as_combined_fn (hi);
> + optab1 = lookup_hilo_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
> + optab2 = lookup_hilo_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
>
> so that tries to automatically associate the scalar widening IFN
> with the set(s) of IFN pairs we can split to. But then this
> list should be static and there's no need to create a std::map?
> Maybe gencfn-macros.cc can be enhanced to output these static
> cases? Or the vectorizer could (as it did previously) simply
> open-code the handled cases (I guess since we deal with two
> cases only now I'd prefer that).
>
> Thanks,
> Richard.
>
>
>> Thanks,
>> Richard
>>
>
The patch I uploaded last no longer has std::map nor
internal_fn_hilo_keys_array and internal_fn_hilo_values_array. (I've
attached it again)
I'm not sure I understand the _halves, do you mean that for the case
where I had _hilo or _HILO before we rename that to _halves/_HALVES such
that it later represents both _hi/_lo separation and _even/_odd?
And am I correct to assume we are just giving up on having a
INTERNAL_OPTAB_FN idea for 1)?
Kind regards,
Andre
[-- Attachment #2: ifn1v3.patch --]
[-- Type: text/plain, Size: 34240 bytes --]
diff --git a/gcc/config/aarch64/aarch64-simd.md b/gcc/config/aarch64/aarch64-simd.md
index bfc98a8d943467b33390defab9682f44efab5907..ffbbecb9409e1c2835d658c2a8855cd0e955c0f2 100644
--- a/gcc/config/aarch64/aarch64-simd.md
+++ b/gcc/config/aarch64/aarch64-simd.md
@@ -4626,7 +4626,7 @@
[(set_attr "type" "neon_<ADDSUB:optab>_long")]
)
-(define_expand "vec_widen_<su>addl_lo_<mode>"
+(define_expand "vec_widen_<su>add_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4638,7 +4638,7 @@
DONE;
})
-(define_expand "vec_widen_<su>addl_hi_<mode>"
+(define_expand "vec_widen_<su>add_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4650,7 +4650,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_lo_<mode>"
+(define_expand "vec_widen_<su>sub_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4662,7 +4662,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_hi_<mode>"
+(define_expand "vec_widen_<su>sub_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index 8b2882da4fe7da07d22b4e5384d049ba7d3907bf..0fd7e6cce8bbd4ecb8027b702722adcf6c32eb55 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,6 +1811,10 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
+@tindex IFN_VEC_WIDEN_PLUS_HI
+@tindex IFN_VEC_WIDEN_PLUS_LO
+@tindex IFN_VEC_WIDEN_MINUS_HI
+@tindex IFN_VEC_WIDEN_MINUS_LO
@tindex VEC_WIDEN_PLUS_HI_EXPR
@tindex VEC_WIDEN_PLUS_LO_EXPR
@tindex VEC_WIDEN_MINUS_HI_EXPR
@@ -1861,6 +1865,33 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
+@item IFN_VEC_WIDEN_PLUS_HI
+@itemx IFN_VEC_WIDEN_PLUS_LO
+These internal functions represent widening vector addition of the high and low
+parts of the two input vectors, respectively. Their operands are vectors that
+contain the same number of elements (@code{N}) of the same integral type. The
+result is a vector that contains half as many elements, of an integral type
+whose size is twice as wide. In the case of @code{IFN_VEC_WIDEN_PLUS_HI} the
+high @code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} products. In the case of @code{IFN_VEC_WIDEN_PLUS_LO} the low
+@code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} products.
+
+@item IFN_VEC_WIDEN_MINUS_HI
+@itemx IFN_VEC_WIDEN_MINUS_LO
+These internal functions represent widening vector subtraction of the high and
+low parts of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The high/low elements of the second vector are subtracted from the high/low
+elements of the first. The result is a vector that contains half as many
+elements, of an integral type whose size is twice as wide. In the case of
+@code{IFN_VEC_WIDEN_MINUS_HI} the high @code{N/2} elements of the second
+vector are subtracted from the high @code{N/2} of the first to produce the
+vector of @code{N/2} products. In the case of
+@code{IFN_VEC_WIDEN_MINUS_LO} the low @code{N/2} elements of the second
+vector are subtracted from the low @code{N/2} of the first to produce the
+vector of @code{N/2} products.
+
@item VEC_WIDEN_PLUS_HI_EXPR
@itemx VEC_WIDEN_PLUS_LO_EXPR
These nodes represent widening vector addition of the high and low parts of
diff --git a/gcc/gimple-range-op.cc b/gcc/gimple-range-op.cc
index 594bd3043f0e944299ddfff219f757ef15a3dd61..66636d82df27626e7911efd0cb8526921b39633f 100644
--- a/gcc/gimple-range-op.cc
+++ b/gcc/gimple-range-op.cc
@@ -1187,6 +1187,7 @@ gimple_range_op_handler::maybe_non_standard ()
{
range_operator *signed_op = ptr_op_widen_mult_signed;
range_operator *unsigned_op = ptr_op_widen_mult_unsigned;
+ bool signed1, signed2, signed_ret;
if (gimple_code (m_stmt) == GIMPLE_ASSIGN)
switch (gimple_assign_rhs_code (m_stmt))
{
@@ -1202,32 +1203,55 @@ gimple_range_op_handler::maybe_non_standard ()
m_op1 = gimple_assign_rhs1 (m_stmt);
m_op2 = gimple_assign_rhs2 (m_stmt);
tree ret = gimple_assign_lhs (m_stmt);
- bool signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
- bool signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
- bool signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
-
- /* Normally these operands should all have the same sign, but
- some passes and violate this by taking mismatched sign args. At
- the moment the only one that's possible is mismatch inputs and
- unsigned output. Once ranger supports signs for the operands we
- can properly fix it, for now only accept the case we can do
- correctly. */
- if ((signed1 ^ signed2) && signed_ret)
- return;
-
- m_valid = true;
- if (signed2 && !signed1)
- std::swap (m_op1, m_op2);
-
- if (signed1 || signed2)
- m_int = signed_op;
- else
- m_int = unsigned_op;
+ signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
+ signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
+ signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
break;
}
default:
- break;
+ return;
}
+ else if (gimple_code (m_stmt) == GIMPLE_CALL
+ && gimple_call_internal_p (m_stmt)
+ && gimple_get_lhs (m_stmt) != NULL_TREE)
+ switch (gimple_call_internal_fn (m_stmt))
+ {
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
+ {
+ signed_op = ptr_op_widen_plus_signed;
+ unsigned_op = ptr_op_widen_plus_unsigned;
+ m_valid = false;
+ m_op1 = gimple_call_arg (m_stmt, 0);
+ m_op2 = gimple_call_arg (m_stmt, 1);
+ tree ret = gimple_get_lhs (m_stmt);
+ signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
+ signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
+ signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
+ break;
+ }
+ default:
+ return;
+ }
+ else
+ return;
+
+ /* Normally these operands should all have the same sign, but some passes
+ and violate this by taking mismatched sign args. At the moment the only
+ one that's possible is mismatch inputs and unsigned output. Once ranger
+ supports signs for the operands we can properly fix it, for now only
+ accept the case we can do correctly. */
+ if ((signed1 ^ signed2) && signed_ret)
+ return;
+
+ m_valid = true;
+ if (signed2 && !signed1)
+ std::swap (m_op1, m_op2);
+
+ if (signed1 || signed2)
+ m_int = signed_op;
+ else
+ m_int = unsigned_op;
}
// Set up a gimple_range_op_handler for any built in function which can be
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 5c9da73ea11f8060b18dcf513599c9694fa4f2ad..1acea5ae33046b70de247b1688aea874d9956abc 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -90,6 +90,19 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+/* Given an internal_fn IFN that is a HILO function, return its corresponding
+ LO and HI internal_fns. */
+
+extern void
+lookup_hilo_internal_fn (internal_fn ifn, internal_fn *lo, internal_fn *hi)
+{
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -137,7 +150,16 @@ const direct_internal_fn_info direct_internal_fn_array[IFN_LAST + 1] = {
#define DEF_INTERNAL_OPTAB_FN(CODE, FLAGS, OPTAB, TYPE) TYPE##_direct,
#define DEF_INTERNAL_SIGNED_OPTAB_FN(CODE, FLAGS, SELECTOR, SIGNED_OPTAB, \
UNSIGNED_OPTAB, TYPE) TYPE##_direct,
+#undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+#define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(CODE, FLAGS, SELECTOR, SIGNED_OPTAB, \
+ UNSIGNED_OPTAB, TYPE) \
+TYPE##_direct, TYPE##_direct, TYPE##_direct,
+#define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(CODE, FLAGS, OPTAB, TYPE) \
+TYPE##_direct, TYPE##_direct, TYPE##_direct,
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
not_direct
};
@@ -3852,7 +3874,7 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types,
/* Return the optab used by internal function FN. */
-static optab
+optab
direct_internal_fn_optab (internal_fn fn, tree_pair types)
{
switch (fn)
@@ -3971,6 +3993,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS_HILO:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -4044,6 +4069,88 @@ first_commutative_argument (internal_fn fn)
}
}
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as wide as the element size of the input vectors. */
+
+bool
+widening_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME##_HILO:\
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as narrow as the element size of the input vectors. */
+
+bool
+narrowing_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(NAME, F, O, T) \
+ case IFN_##NAME##_HILO:\
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if FN decomposes to _hi and _lo IFN. */
+
+bool
+decomposes_to_hilo_fn_p (internal_fn fn)
+{
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME##_HILO:\
+ return true;
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+ #define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(NAME, F, O, T) \
+ case IFN_##NAME##_HILO:\
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+ #undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+
+ default:
+ return false;
+ }
+}
+
/* Return true if IFN_SET_EDOM is supported. */
bool
@@ -4071,7 +4178,33 @@ set_edom_supported_p (void)
optab which_optab = direct_internal_fn_optab (fn, types); \
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
+#define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(CODE, FLAGS, SELECTOR, \
+ SIGNED_OPTAB, UNSIGNED_OPTAB, \
+ TYPE) \
+ static void \
+ expand_##CODE##_HILO (internal_fn fn ATTRIBUTE_UNUSED, \
+ gcall *stmt ATTRIBUTE_UNUSED) \
+ { \
+ gcc_unreachable (); \
+ } \
+ DEF_INTERNAL_SIGNED_OPTAB_FN(CODE##_HI, FLAGS, SELECTOR, SIGNED_OPTAB, \
+ UNSIGNED_OPTAB, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN(CODE##_LO, FLAGS, SELECTOR, SIGNED_OPTAB, \
+ UNSIGNED_OPTAB, TYPE)
+#define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(CODE, FLAGS, OPTAB, TYPE) \
+ static void \
+ expand_##CODE##_HILO (internal_fn fn ATTRIBUTE_UNUSED, \
+ gcall *stmt ATTRIBUTE_UNUSED) \
+ { \
+ gcc_unreachable (); \
+ } \
+ DEF_INTERNAL_OPTAB_FN(CODE##_LO, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN(CODE##_HI, FLAGS, OPTAB, TYPE)
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_FN
+#undef DEF_INTERNAL_SIGNED_OPTAB_FN
+#undef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#undef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
@@ -4080,6 +4213,7 @@ set_edom_supported_p (void)
where STMT is the statement that performs the call. */
static void (*const internal_fn_expanders[]) (internal_fn, gcall *) = {
+
#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) expand_##CODE,
#include "internal-fn.def"
0
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 7fe742c2ae713e7152ab05cfdfba86e4e0aa3456..012dd323b86dd7cfcc5c13d3a2bb2a453937155d 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -85,6 +85,13 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_SIGNED_OPTAB_HILO_FN is like DEF_INTERNAL_OPTAB_FN except it
+ provides convenience wrappers for defining conversions that require a
+ hi/lo split, like widening and narrowing operations. Each definition
+ for <NAME> will require an optab named <OPTAB> and two other optabs that
+ you specify for signed and unsigned.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -123,6 +130,20 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_OPTAB_WIDENING_HILO_FN
+#define DEF_INTERNAL_OPTAB_WIDENING_HILO_FN(NAME, FLAGS, SELECTOR, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_FN (NAME##_HILO, FLAGS | ECF_LEAF, NULL) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _LO, FLAGS, SELECTOR, SOPTAB##_lo, UOPTAB##_lo, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _HI, FLAGS, SELECTOR, SOPTAB##_hi, UOPTAB##_hi, TYPE)
+#endif
+
+#ifndef DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
+#define DEF_INTERNAL_OPTAB_NARROWING_HILO_FN(NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_FN (NAME##_HILO, FLAGS | ECF_LEAF, NULL) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, OPTAB##_lo, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, OPTAB##_hi, TYPE)
+#endif
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -315,6 +336,16 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_OPTAB_WIDENING_HILO_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_sadd, vec_widen_uadd,
+ binary)
+DEF_INTERNAL_OPTAB_WIDENING_HILO_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_ssub, vec_widen_usub,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 08922ed4254898f5fffca3f33973e96ed9ce772f..8ba07d6d1338e75bc5a451d9e403112a608f3ea2 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,8 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern void lookup_hilo_internal_fn (internal_fn, internal_fn *, internal_fn *);
+extern optab direct_internal_fn_optab (internal_fn, tree_pair);
/* Return the ECF_* flags for function FN. */
@@ -210,6 +216,9 @@ extern bool commutative_binary_fn_p (internal_fn);
extern bool commutative_ternary_fn_p (internal_fn);
extern int first_commutative_argument (internal_fn);
extern bool associative_binary_fn_p (internal_fn);
+extern bool widening_fn_p (code_helper);
+extern bool narrowing_fn_p (code_helper);
+extern bool decomposes_to_hilo_fn_p (internal_fn);
extern bool set_edom_supported_p (void);
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index c8e39c82d57a7d726e7da33d247b80f32ec9236c..5a08d91e550b2d92e9572211f811fdba99a33a38 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,15 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab
+ || binoptab == vec_widen_sadd_hi_optab
+ || binoptab == vec_widen_sadd_lo_optab
+ || binoptab == vec_widen_uadd_hi_optab
+ || binoptab == vec_widen_uadd_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 695f5911b300c9ca5737de9be809fa01aabe5e01..16d121722c8c5723d9b164f5a2c616dc7ec143de 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -410,6 +410,10 @@ OPTAB_D (vec_widen_ssubl_hi_optab, "vec_widen_ssubl_hi_$a")
OPTAB_D (vec_widen_ssubl_lo_optab, "vec_widen_ssubl_lo_$a")
OPTAB_D (vec_widen_saddl_hi_optab, "vec_widen_saddl_hi_$a")
OPTAB_D (vec_widen_saddl_lo_optab, "vec_widen_saddl_lo_$a")
+OPTAB_D (vec_widen_ssub_hi_optab, "vec_widen_ssub_hi_$a")
+OPTAB_D (vec_widen_ssub_lo_optab, "vec_widen_ssub_lo_$a")
+OPTAB_D (vec_widen_sadd_hi_optab, "vec_widen_sadd_hi_$a")
+OPTAB_D (vec_widen_sadd_lo_optab, "vec_widen_sadd_lo_$a")
OPTAB_D (vec_widen_sshiftl_hi_optab, "vec_widen_sshiftl_hi_$a")
OPTAB_D (vec_widen_sshiftl_lo_optab, "vec_widen_sshiftl_lo_$a")
OPTAB_D (vec_widen_umult_even_optab, "vec_widen_umult_even_$a")
@@ -422,6 +426,10 @@ OPTAB_D (vec_widen_usubl_hi_optab, "vec_widen_usubl_hi_$a")
OPTAB_D (vec_widen_usubl_lo_optab, "vec_widen_usubl_lo_$a")
OPTAB_D (vec_widen_uaddl_hi_optab, "vec_widen_uaddl_hi_$a")
OPTAB_D (vec_widen_uaddl_lo_optab, "vec_widen_uaddl_lo_$a")
+OPTAB_D (vec_widen_usub_hi_optab, "vec_widen_usub_hi_$a")
+OPTAB_D (vec_widen_usub_lo_optab, "vec_widen_usub_lo_$a")
+OPTAB_D (vec_widen_uadd_hi_optab, "vec_widen_uadd_hi_$a")
+OPTAB_D (vec_widen_uadd_lo_optab, "vec_widen_uadd_lo_$a")
OPTAB_D (vec_addsub_optab, "vec_addsub$a3")
OPTAB_D (vec_fmaddsub_optab, "vec_fmaddsub$a4")
OPTAB_D (vec_fmsubadd_optab, "vec_fmsubadd$a4")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index 0aeebb67fac864db284985f4a6f0653af281d62b..28464ad9e3a7ea25557ffebcdbdbc1340f9e0d8b 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -65,6 +65,7 @@ along with GCC; see the file COPYING3. If not see
#include "asan.h"
#include "profile.h"
#include "sreal.h"
+#include "internal-fn.h"
/* This file contains functions for building the Control Flow Graph (CFG)
for a function tree. */
@@ -3411,6 +3412,52 @@ verify_gimple_call (gcall *stmt)
debug_generic_stmt (fn);
return true;
}
+ internal_fn ifn = gimple_call_internal_fn (stmt);
+ if (ifn == IFN_LAST)
+ {
+ error ("gimple call has an invalid IFN");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ else if (decomposes_to_hilo_fn_p (ifn))
+ {
+ /* Non decomposed HILO stmts should not appear in IL, these are
+ merely used as an internal representation to the auto-vectorizer
+ pass and should have been expanded to their _LO _HI variants. */
+ error ("gimple call has an non decomposed HILO IFN");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ else if (ifn == IFN_VEC_WIDEN_PLUS_LO
+ || ifn == IFN_VEC_WIDEN_PLUS_HI
+ || ifn == IFN_VEC_WIDEN_MINUS_LO
+ || ifn == IFN_VEC_WIDEN_MINUS_HI)
+ {
+ tree rhs1_type = TREE_TYPE (gimple_call_arg (stmt, 0));
+ tree rhs2_type = TREE_TYPE (gimple_call_arg (stmt, 1));
+ tree lhs_type = TREE_TYPE (gimple_get_lhs (stmt));
+ if (TREE_CODE (lhs_type) == VECTOR_TYPE)
+ {
+ if (TREE_CODE (rhs1_type) != VECTOR_TYPE
+ || TREE_CODE (rhs2_type) != VECTOR_TYPE)
+ {
+ error ("invalid non-vector operands in vector IFN call");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ lhs_type = TREE_TYPE (lhs_type);
+ rhs1_type = TREE_TYPE (rhs1_type);
+ rhs2_type = TREE_TYPE (rhs2_type);
+ }
+ if (POINTER_TYPE_P (lhs_type)
+ || POINTER_TYPE_P (rhs1_type)
+ || POINTER_TYPE_P (rhs2_type))
+ {
+ error ("invalid (pointer) operands in vector IFN call");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ }
}
else
{
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 63a19f8d1d89c6bd5d8e55a299cbffaa324b4b84..d74d8db2173b1ab117250fea89de5212d5e354ec 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4433,7 +4433,20 @@ estimate_num_insns (gimple *stmt, eni_weights *weights)
tree decl;
if (gimple_call_internal_p (stmt))
- return 0;
+ {
+ internal_fn fn = gimple_call_internal_fn (stmt);
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS_HI:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_MINUS_HI:
+ case IFN_VEC_WIDEN_MINUS_LO:
+ return 1;
+
+ default:
+ return 0;
+ }
+ }
else if ((decl = gimple_call_fndecl (stmt))
&& fndecl_built_in_p (decl))
{
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 1778af0242898e3dc73d94d22a5b8505628a53b5..93cebc72beb4f65249a69b2665dfeb8a0991c1d1 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -562,21 +562,30 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
+ return 0;
+
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else if (is_gimple_call (stmt))
+ rhs_code = gimple_call_combined_fn (stmt);
+ else
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ if (rhs_code != code
+ && rhs_code != widened_code)
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = gimple_get_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -589,7 +598,7 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op = gimple_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1343,7 +1352,8 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ IFN_VEC_WIDEN_MINUS_HILO,
false, 2, unprom, &half_type))
return NULL;
@@ -1395,14 +1405,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1468,6 +1480,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1481,26 +1507,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS_HILO. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS_HILO,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS_HILO. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS_HILO,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_ctz_ffs_pattern
@@ -3078,7 +3108,7 @@ vect_recog_average_pattern (vec_info *vinfo,
vect_unpromoted_value unprom[3];
tree new_type;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
+ IFN_VEC_WIDEN_PLUS_HILO, false, 3,
unprom, &new_type);
if (nops == 0)
return NULL;
@@ -6469,6 +6499,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index d152ae9ab10b361b88c0f839d6951c43b954750a..24c811ebe01fb8b003100dea494cf64fea72a975 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5038,7 +5038,9 @@ vectorizable_conversion (vec_info *vinfo,
bool widen_arith = (code == WIDEN_PLUS_EXPR
|| code == WIDEN_MINUS_EXPR
|| code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
+ || code == WIDEN_LSHIFT_EXPR
+ || code == IFN_VEC_WIDEN_PLUS_HILO
+ || code == IFN_VEC_WIDEN_MINUS_HILO);
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -5088,7 +5090,9 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
+ || code == WIDEN_MINUS_EXPR
+ || code == IFN_VEC_WIDEN_PLUS_HILO
+ || code == IFN_VEC_WIDEN_MINUS_HILO);
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
@@ -12478,10 +12482,43 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
+
+ if (code.is_fn_code ())
+ {
+ internal_fn ifn = as_internal_fn ((combined_fn) code);
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ internal_fn lo, hi;
+ lookup_hilo_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = direct_internal_fn_optab (lo, {vectype, vectype});
+ optab2 = direct_internal_fn_optab (hi, {vectype, vectype});
+ }
+ else if (code.is_tree_code ())
{
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P ((tree_code) code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1, vectype, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ }
}
if (!optab1 || !optab2)
diff --git a/gcc/tree.def b/gcc/tree.def
index 90ceeec0b512bfa5f983359c0af03cc71de32007..b37b0b35927b92a6536e5c2d9805ffce8319a240 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1374,15 +1374,16 @@ DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr", tcc_expression, 3)
DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr", tcc_binary, 2)
/* Widening sad (sum of absolute differences).
- The first two arguments are of type t1 which should be integer.
- The third argument and the result are of type t2, such that t2 is at least
- twice the size of t1. Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
+ The first two arguments are of type t1 which should be a vector of integers.
+ The third argument and the result are of type t2, such that the size of
+ the elements of t2 is at least twice the size of the elements of t1.
+ Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
equivalent to:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = PLUS_EXPR (tmp2, arg3)
or:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = WIDEN_SUM_EXPR (tmp2, arg3)
*/
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-15 11:53 ` Andre Vieira (lists)
@ 2023-05-15 12:21 ` Richard Biener
2023-05-18 17:15 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-05-15 12:21 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Sandiford, Richard Biener, gcc-patches
On Mon, 15 May 2023, Andre Vieira (lists) wrote:
>
>
> On 15/05/2023 12:01, Richard Biener wrote:
> > On Mon, 15 May 2023, Richard Sandiford wrote:
> >
> >> Richard Biener <rguenther@suse.de> writes:
> >>> On Fri, 12 May 2023, Richard Sandiford wrote:
> >>>
> >>>> Richard Biener <rguenther@suse.de> writes:
> >>>>> On Fri, 12 May 2023, Andre Vieira (lists) wrote:
> >>>>>
> >>>>>> I have dealt with, I think..., most of your comments. There's quite a
> >>>>>> few
> >>>>>> changes, I think it's all a bit simpler now. I made some other changes
> >>>>>> to the
> >>>>>> costing in tree-inline.cc and gimple-range-op.cc in which I try to
> >>>>>> preserve
> >>>>>> the same behaviour as we had with the tree codes before. Also added
> >>>>>> some extra
> >>>>>> checks to tree-cfg.cc that made sense to me.
> >>>>>>
> >>>>>> I am still regression testing the gimple-range-op change, as that was a
> >>>>>> last
> >>>>>> minute change, but the rest survived a bootstrap and regression test on
> >>>>>> aarch64-unknown-linux-gnu.
> >>>>>>
> >>>>>> cover letter:
> >>>>>>
> >>>>>> This patch replaces the existing tree_code widen_plus and widen_minus
> >>>>>> patterns with internal_fn versions.
> >>>>>>
> >>>>>> DEF_INTERNAL_OPTAB_WIDENING_HILO_FN and
> >>>>>> DEF_INTERNAL_OPTAB_NARROWING_HILO_FN
> >>>>>> are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN
> >>>>>> respectively
> >>>>>> except they provide convenience wrappers for defining conversions that
> >>>>>> require
> >>>>>> a hi/lo split. Each definition for <NAME> will require optabs for _hi
> >>>>>> and _lo
> >>>>>> and each of those will also require a signed and unsigned version in
> >>>>>> the case
> >>>>>> of widening. The hi/lo pair is necessary because the widening and
> >>>>>> narrowing
> >>>>>> operations take n narrow elements as inputs and return n/2 wide
> >>>>>> elements as
> >>>>>> outputs. The 'lo' operation operates on the first n/2 elements of
> >>>>>> input. The
> >>>>>> 'hi' operation operates on the second n/2 elements of input. Defining
> >>>>>> an
> >>>>>> internal_fn along with hi/lo variations allows a single internal
> >>>>>> function to
> >>>>>> be returned from a vect_recog function that will later be expanded to
> >>>>>> hi/lo.
> >>>>>>
> >>>>>>
> >>>>>> For example:
> >>>>>> IFN_VEC_WIDEN_PLUS -> IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO
> >>>>>> for aarch64: IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> ->
> >>>>>> (u/s)addl2
> >>>>>> IFN_VEC_WIDEN_PLUS_LO ->
> >>>>>> vec_widen_<su>add_lo_<mode>
> >>>>>> -> (u/s)addl
> >>>>>>
> >>>>>> This gives the same functionality as the previous
> >>>>>> WIDEN_PLUS/WIDEN_MINUS tree
> >>>>>> codes which are expanded into VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
> >>>>>
> >>>>> What I still don't understand is how we are so narrowly focused on
> >>>>> HI/LO? We need a combined scalar IFN for pattern selection (not
> >>>>> sure why that's now called _HILO, I expected no suffix). Then there's
> >>>>> three possibilities the target can implement this:
> >>>>>
> >>>>> 1) with a widen_[su]add<mode> instruction - I _think_ that's what
> >>>>> RISCV is going to offer since it is a target where vector modes
> >>>>> have "padding" (aka you cannot subreg a V2SI to get V4HI). Instead
> >>>>> RVV can do a V4HI to V4SI widening and widening add/subtract
> >>>>> using vwadd[u] and vwsub[u] (the HI->SI widening is actually
> >>>>> done with a widening add of zero - eh).
> >>>>> IIRC GCN is the same here.
> >>>>
> >>>> SVE currently does this too, but the addition and widening are
> >>>> separate operations. E.g. in principle there's no reason why
> >>>> you can't sign-extend one operand, zero-extend the other, and
> >>>> then add the result together. Or you could extend them from
> >>>> different sizes (QI and HI). All of those are supported
> >>>> (if the costing allows them).
> >>>
> >>> I see. So why does the target the expose widen_[su]add<mode> at all?
> >>
> >> It shouldn't (need to) do that. I don't think we should have an optab
> >> for the unsplit operation.
> >>
> >> At least on SVE, we really want the extensions to be fused with loads
> >> (where possible) rather than with arithmetic.
> >>
> >> We can still do the widening arithmetic in one go. It's just that
> >> fusing with the loads works for the mixed-sign and mixed-size cases,
> >> and can handle more than just doubling the element size.
> >>
> >>>> If the target has operations to do combined extending and adding (or
> >>>> whatever), then at the moment we rely on combine to generate them.
> >>>>
> >>>> So I think this case is separate from Andre's work. The addition
> >>>> itself is just an ordinary addition, and any widening happens by
> >>>> vectorising a CONVERT/NOP_EXPR.
> >>>>
> >>>>> 2) with a widen_[su]add{_lo,_hi}<mode> combo - that's what the tree
> >>>>> codes currently support (exclusively)
> >>>>> 3) similar, but widen_[su]add{_even,_odd}<mode>
> >>>>>
> >>>>> that said, things like decomposes_to_hilo_fn_p look to paint us into
> >>>>> a 2) corner without good reason.
> >>>>
> >>>> I suppose one question is: how much of the patch is really specific
> >>>> to HI/LO, and how much is just grouping two halves together?
> >>>
> >>> Yep, that I don't know for sure.
> >>>
> >>>> The nice
> >>>> thing about the internal-fn grouping macros is that, if (3) is
> >>>> implemented in future, the structure will strongly encourage even/odd
> >>>> pairs to be supported for all operations that support hi/lo. That is,
> >>>> I would expect the grouping macros to be extended to define even/odd
> >>>> ifns alongside hi/lo ones, rather than adding separate definitions
> >>>> for even/odd functions.
> >>>>
> >>>> If so, at least from the internal-fn.* side of things, I think the
> >>>> question
> >>>> is whether it's OK to stick with hilo names for now, or whether we should
> >>>> use more forward-looking names.
> >>>
> >>> I think for parts that are independent we could use a more
> >>> forward-looking name. Maybe _halves?
> >>
> >> Using _halves for the ifn macros sounds good to me FWIW.
> >>
> >>> But I'm also not sure
> >>> how much of that is really needed (it seems to be tied around
> >>> optimizing optabs space?)
> >>
> >> Not sure what you mean by "this". Optabs space shouldn't be a problem
> >> though. The optab encoding gives us a full int to play with, and it
> >> could easily go up to 64 bits if necessary/convenient.
> >>
> >> At least on the internal-fn.* side, the aim is really just to establish
> >> a regular structure, so that we don't have arbitrary differences between
> >> different widening operations, or too much cut-&-paste.
> >
> > Hmm, I'm looking at the need for the std::map and
> > internal_fn_hilo_keys_array and internal_fn_hilo_values_array.
> > The vectorizer pieces contain
> >
> > + if (code.is_fn_code ())
> > + {
> > + internal_fn ifn = as_internal_fn ((combined_fn) code);
> > + gcc_assert (decomposes_to_hilo_fn_p (ifn));
> > +
> > + internal_fn lo, hi;
> > + lookup_hilo_internal_fn (ifn, &lo, &hi);
> > + *code1 = as_combined_fn (lo);
> > + *code2 = as_combined_fn (hi);
> > + optab1 = lookup_hilo_ifn_optab (lo, !TYPE_UNSIGNED (vectype));
> > + optab2 = lookup_hilo_ifn_optab (hi, !TYPE_UNSIGNED (vectype));
> >
> > so that tries to automatically associate the scalar widening IFN
> > with the set(s) of IFN pairs we can split to. But then this
> > list should be static and there's no need to create a std::map?
> > Maybe gencfn-macros.cc can be enhanced to output these static
> > cases? Or the vectorizer could (as it did previously) simply
> > open-code the handled cases (I guess since we deal with two
> > cases only now I'd prefer that).
> >
> > Thanks,
> > Richard.
> >
> >
> >> Thanks,
> >> Richard
> >>
> >
> The patch I uploaded last no longer has std::map nor
> internal_fn_hilo_keys_array and internal_fn_hilo_values_array. (I've attached
> it again)
Whoops, too many patches ...
> I'm not sure I understand the _halves, do you mean that for the case where I
> had _hilo or _HILO before we rename that to _halves/_HALVES such that it later
> represents both _hi/_lo separation and _even/_odd?
I don't see much shared stuff, but I guess we'd see when we add a case
for EVEN/ODD. The verifier contains
+ else if (decomposes_to_hilo_fn_p (ifn))
+ {
+ /* Non decomposed HILO stmts should not appear in IL, these are
+ merely used as an internal representation to the
auto-vectorizer
+ pass and should have been expanded to their _LO _HI variants.
*/
+ error ("gimple call has an non decomposed HILO IFN");
+ debug_generic_stmt (fn);
+ return true;
I think to support case 1) that's not wanted. Instead what you could
check is that the types involved are vector types, so a subset of
what you check for IFN_VEC_WIDEN_PLUS_LO etc. (but oddly it's not
verified those are all operating on vector types only?)
+/* Given an internal_fn IFN that is a HILO function, return its
corresponding
+ LO and HI internal_fns. */
+
+extern void
+lookup_hilo_internal_fn (internal_fn ifn, internal_fn *lo, internal_fn
*hi)
+{
+ gcc_assert (decomposes_to_hilo_fn_p (ifn));
+
+ *lo = internal_fn (ifn + 1);
+ *hi = internal_fn (ifn + 2);
that might become fragile if we add EVEN/ODD besides HI/LO unless
we merge those with a DEF_INTERNAL_OPTAB_WIDENING_HILO_EVENODD_FN
case, right?
> And am I correct to assume we are just giving up on having a INTERNAL_OPTAB_FN
> idea for 1)?
Well, I think we want all of them in the end (or at least support them
if target need arises). full vector, hi/lo and even/odd.
Richard.
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-15 12:21 ` Richard Biener
@ 2023-05-18 17:15 ` Andre Vieira (lists)
2023-05-22 13:06 ` Richard Biener
0 siblings, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-05-18 17:15 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Sandiford, Richard Biener, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 2935 bytes --]
How about this?
Not sure about the DEF_INTERNAL documentation I rewrote in
internal-fn.def, was struggling to word these, so improvements welcome!
gcc/ChangeLog:
2023-04-25 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
Tamar Christina <tamar.christina@arm.com>
* config/aarch64/aarch64-simd.md
(vec_widen_<su>addl_lo_<mode>): Rename
this ...
(vec_widen_<su>add_lo_<mode>): ... to this.
(vec_widen_<su>addl_hi_<mode>): Rename this ...
(vec_widen_<su>add_hi_<mode>): ... to this.
(vec_widen_<su>subl_lo_<mode>): Rename this ...
(vec_widen_<su>sub_lo_<mode>): ... to this.
(vec_widen_<su>subl_hi_<mode>): Rename this ...
(vec_widen_<su>sub_hi_<mode>): ...to this.
* doc/generic.texi: Document new IFN codes.
* internal-fn.cc (ifn_cmp): Function to compare ifn's for
sorting/searching.
(lookup_hilo_internal_fn): Add lookup function.
(commutative_binary_fn_p): Add widen_plus fn's.
(widening_fn_p): New function.
(narrowing_fn_p): New function.
(direct_internal_fn_optab): Change visibility.
* internal-fn.def (DEF_INTERNAL_WIDENING_OPTAB_FN): Macro to define an
internal_fn that expands into multiple internal_fns for widening.
(DEF_INTERNAL_NARROWING_OPTAB_FN): Likewise but for narrowing.
(IFN_VEC_WIDEN_PLUS, IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO,
IFN_VEC_WIDEN_PLUS_EVEN, IFN_VEC_WIDEN_PLUS_ODD,
IFN_VEC_WIDEN_MINUS, IFN_VEC_WIDEN_MINUS_HI,
IFN_VEC_WIDEN_MINUS_LO,
IFN_VEC_WIDEN_MINUS_ODD, IFN_VEC_WIDEN_MINUS_EVEN): Define
widening
plus,minus functions.
* internal-fn.h (direct_internal_fn_optab): Declare new prototype.
(lookup_hilo_internal_fn): Likewise.
(widening_fn_p): Likewise.
(Narrowing_fn_p): Likewise.
* optabs.cc (commutative_optab_p): Add widening plus optabs.
* optabs.def (OPTAB_D): Define widen add, sub optabs.
* tree-cfg.cc (verify_gimple_call): Add checks for widening ifns.
* tree-inline.cc (estimate_num_insns): Return same
cost for widen add and sub IFNs as previous tree_codes.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
patterns with a hi/lo or even/odd split.
(vect_recog_sad_pattern): Refactor to use new IFN codes.
(vect_recog_widen_plus_pattern): Likewise.
(vect_recog_widen_minus_pattern): Likewise.
(vect_recog_average_pattern): Likewise.
* tree-vect-stmts.cc (vectorizable_conversion): Add support for
_HILO IFNs.
(supportable_widening_operation): Likewise.
* tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-widen-add.c: Test that new
IFN_VEC_WIDEN_PLUS is being used.
* gcc.target/aarch64/vect-widen-sub.c: Test that new
IFN_VEC_WIDEN_MINUS is being used.
[-- Attachment #2: ifn1v4.patch --]
[-- Type: text/plain, Size: 39515 bytes --]
diff --git a/gcc/config/aarch64/aarch64-simd.md b/gcc/config/aarch64/aarch64-simd.md
index bfc98a8d943467b33390defab9682f44efab5907..ffbbecb9409e1c2835d658c2a8855cd0e955c0f2 100644
--- a/gcc/config/aarch64/aarch64-simd.md
+++ b/gcc/config/aarch64/aarch64-simd.md
@@ -4626,7 +4626,7 @@
[(set_attr "type" "neon_<ADDSUB:optab>_long")]
)
-(define_expand "vec_widen_<su>addl_lo_<mode>"
+(define_expand "vec_widen_<su>add_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4638,7 +4638,7 @@
DONE;
})
-(define_expand "vec_widen_<su>addl_hi_<mode>"
+(define_expand "vec_widen_<su>add_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4650,7 +4650,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_lo_<mode>"
+(define_expand "vec_widen_<su>sub_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4662,7 +4662,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_hi_<mode>"
+(define_expand "vec_widen_<su>sub_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index 8b2882da4fe7da07d22b4e5384d049ba7d3907bf..5e36dac2b1a10257616f12cdfb0b12d0f2879ae9 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,10 +1811,16 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
+@tindex IFN_VEC_WIDEN_PLUS
+@tindex IFN_VEC_WIDEN_PLUS_HI
+@tindex IFN_VEC_WIDEN_PLUS_LO
+@tindex IFN_VEC_WIDEN_PLUS_EVEN
+@tindex IFN_VEC_WIDEN_PLUS_ODD
+@tindex IFN_VEC_WIDEN_MINUS
+@tindex IFN_VEC_WIDEN_MINUS_HI
+@tindex IFN_VEC_WIDEN_MINUS_LO
+@tindex IFN_VEC_WIDEN_MINUS_EVEN
+@tindex IFN_VEC_WIDEN_MINUS_ODD
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1861,6 +1867,82 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
+@item IFN_VEC_WIDEN_PLUS
+This internal function represents widening vector addition of two input
+vectors. Its operands are vectors that contain the same number of elements
+(@code{N}) of the same integral type. The result is a vector that contains
+the same amount (@code{N}) of elements, of an integral type whose size is twice
+as wide, as the input vectors. If the current target does not implement the
+corresponding optabs the vectorizer may choose to split it into either a pair
+of @code{IFN_VEC_WIDEN_PLUS_HI} and @code{IFN_VEC_WIDEN_PLUS_LO} or
+@code{IFN_VEC_WIDEN_PLUS_EVEN} and @code{IFN_VEC_WIDEN_PLUS_ODD}, depending
+on what optabs the target implements.
+
+@item IFN_VEC_WIDEN_PLUS_HI
+@itemx IFN_VEC_WIDEN_PLUS_LO
+These internal functions represent widening vector addition of the high and low
+parts of the two input vectors, respectively. Their operands are vectors that
+contain the same number of elements (@code{N}) of the same integral type. The
+result is a vector that contains half as many elements, of an integral type
+whose size is twice as wide. In the case of @code{IFN_VEC_WIDEN_PLUS_HI} the
+high @code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions. In the case of @code{IFN_VEC_WIDEN_PLUS_LO} the low
+@code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions.
+
+@item IFN_VEC_WIDEN_PLUS_EVEN
+@itemx IFN_VEC_WIDEN_PLUS_ODD
+These internal functions represent widening vector addition of the even and odd
+elements of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The result is a vector that contains half as many elements, of an integral type
+whose size is twice as wide. In the case of @code{IFN_VEC_WIDEN_PLUS_EVEN} the
+even @code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions. In the case of @code{IFN_VEC_WIDEN_PLUS_ODD} the odd
+@code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions.
+
+@item IFN_VEC_WIDEN_MINUS
+This internal function represents widening vector subtraction of two input
+vectors. Its operands are vectors that contain the same number of elements
+(@code{N}) of the same integral type. The result is a vector that contains
+the same amount (@code{N}) of elements, of an integral type whose size is twice
+as wide, as the input vectors. If the current target does not implement the
+corresponding optabs the vectorizer may choose to split it into either a pair
+of @code{IFN_VEC_WIDEN_MINUS_HI} and @code{IFN_VEC_WIDEN_MINUS_LO} or
+@code{IFN_VEC_WIDEN_MINUS_EVEN} and @code{IFN_VEC_WIDEN_MINUS_ODD}, depending
+on what optabs the target implements.
+
+@item IFN_VEC_WIDEN_MINUS_HI
+@itemx IFN_VEC_WIDEN_MINUS_LO
+These internal functions represent widening vector subtraction of the high and
+low parts of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The high/low elements of the second vector are subtracted from the high/low
+elements of the first. The result is a vector that contains half as many
+elements, of an integral type whose size is twice as wide. In the case of
+@code{IFN_VEC_WIDEN_MINUS_HI} the high @code{N/2} elements of the second
+vector are subtracted from the high @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions. In the case of
+@code{IFN_VEC_WIDEN_MINUS_LO} the low @code{N/2} elements of the second
+vector are subtracted from the low @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions.
+
+@item IFN_VEC_WIDEN_MINUS_EVEN
+@itemx IFN_VEC_WIDEN_MINUS_ODD
+These internal functions represent widening vector subtraction of the even and
+odd parts of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The even/odd elements of the second vector are subtracted from the even/odd
+elements of the first. The result is a vector that contains half as many
+elements, of an integral type whose size is twice as wide. In the case of
+@code{IFN_VEC_WIDEN_MINUS_EVEN} the even @code{N/2} elements of the second
+vector are subtracted from the even @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions. In the case of
+@code{IFN_VEC_WIDEN_MINUS_ODD} the odd @code{N/2} elements of the second
+vector are subtracted from the odd @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions.
+
@item VEC_WIDEN_PLUS_HI_EXPR
@itemx VEC_WIDEN_PLUS_LO_EXPR
These nodes represent widening vector addition of the high and low parts of
diff --git a/gcc/gimple-range-op.cc b/gcc/gimple-range-op.cc
index 594bd3043f0e944299ddfff219f757ef15a3dd61..33f4b7064a2a22aad49f27b24b409e91a5b89c69 100644
--- a/gcc/gimple-range-op.cc
+++ b/gcc/gimple-range-op.cc
@@ -1187,6 +1187,7 @@ gimple_range_op_handler::maybe_non_standard ()
{
range_operator *signed_op = ptr_op_widen_mult_signed;
range_operator *unsigned_op = ptr_op_widen_mult_unsigned;
+ bool signed1, signed2, signed_ret;
if (gimple_code (m_stmt) == GIMPLE_ASSIGN)
switch (gimple_assign_rhs_code (m_stmt))
{
@@ -1202,32 +1203,55 @@ gimple_range_op_handler::maybe_non_standard ()
m_op1 = gimple_assign_rhs1 (m_stmt);
m_op2 = gimple_assign_rhs2 (m_stmt);
tree ret = gimple_assign_lhs (m_stmt);
- bool signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
- bool signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
- bool signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
-
- /* Normally these operands should all have the same sign, but
- some passes and violate this by taking mismatched sign args. At
- the moment the only one that's possible is mismatch inputs and
- unsigned output. Once ranger supports signs for the operands we
- can properly fix it, for now only accept the case we can do
- correctly. */
- if ((signed1 ^ signed2) && signed_ret)
- return;
-
- m_valid = true;
- if (signed2 && !signed1)
- std::swap (m_op1, m_op2);
-
- if (signed1 || signed2)
- m_int = signed_op;
- else
- m_int = unsigned_op;
+ signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
+ signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
+ signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
break;
}
default:
- break;
+ return;
+ }
+ else if (gimple_code (m_stmt) == GIMPLE_CALL
+ && gimple_call_internal_p (m_stmt)
+ && gimple_get_lhs (m_stmt) != NULL_TREE)
+ switch (gimple_call_internal_fn (m_stmt))
+ {
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
+ {
+ signed_op = ptr_op_widen_plus_signed;
+ unsigned_op = ptr_op_widen_plus_unsigned;
+ m_valid = false;
+ m_op1 = gimple_call_arg (m_stmt, 0);
+ m_op2 = gimple_call_arg (m_stmt, 1);
+ tree ret = gimple_get_lhs (m_stmt);
+ signed1 = TYPE_SIGN (TREE_TYPE (m_op1)) == SIGNED;
+ signed2 = TYPE_SIGN (TREE_TYPE (m_op2)) == SIGNED;
+ signed_ret = TYPE_SIGN (TREE_TYPE (ret)) == SIGNED;
+ break;
+ }
+ default:
+ return;
}
+ else
+ return;
+
+ /* Normally these operands should all have the same sign, but some passes
+ and violate this by taking mismatched sign args. At the moment the only
+ one that's possible is mismatch inputs and unsigned output. Once ranger
+ supports signs for the operands we can properly fix it, for now only
+ accept the case we can do correctly. */
+ if ((signed1 ^ signed2) && signed_ret)
+ return;
+
+ m_valid = true;
+ if (signed2 && !signed1)
+ std::swap (m_op1, m_op2);
+
+ if (signed1 || signed2)
+ m_int = signed_op;
+ else
+ m_int = unsigned_op;
}
// Set up a gimple_range_op_handler for any built in function which can be
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 5c9da73ea11f8060b18dcf513599c9694fa4f2ad..348bee35a35ae4ed9a8652f5349f430c2733e1cb 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -90,6 +90,71 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+/* Given an internal_fn IFN that is either a widening or narrowing function, return its
+ corresponding LO and HI internal_fns. */
+
+extern void
+lookup_hilo_internal_fn (internal_fn ifn, internal_fn *lo, internal_fn *hi)
+{
+ gcc_assert (widening_fn_p (ifn) || narrowing_fn_p (ifn));
+
+ switch (ifn)
+ {
+ default:
+ gcc_unreachable ();
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+#define DEF_INTERNAL_FN(NAME, FLAGS, TYPE)
+#define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME: \
+ *lo = internal_fn (IFN_##NAME##_LO); \
+ *hi = internal_fn (IFN_##NAME##_HI); \
+ break;
+#define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, F, O, T) \
+ case IFN_##NAME: \
+ *lo = internal_fn (IFN_##NAME##_LO); \
+ *hi = internal_fn (IFN_##NAME##_HI); \
+ break;
+#include "internal-fn.def"
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+ }
+}
+
+extern void
+lookup_evenodd_internal_fn (internal_fn ifn, internal_fn *even,
+ internal_fn *odd)
+{
+ gcc_assert (widening_fn_p (ifn) || narrowing_fn_p (ifn));
+
+ switch (ifn)
+ {
+ default:
+ gcc_unreachable ();
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+#define DEF_INTERNAL_FN(NAME, FLAGS, TYPE)
+#define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME: \
+ *even = internal_fn (IFN_##NAME##_EVEN); \
+ *odd = internal_fn (IFN_##NAME##_ODD); \
+ break;
+#define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, F, O, T) \
+ case IFN_##NAME: \
+ *even = internal_fn (IFN_##NAME##_EVEN); \
+ *odd = internal_fn (IFN_##NAME##_ODD); \
+ break;
+#include "internal-fn.def"
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+ }
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -3852,7 +3917,7 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types,
/* Return the optab used by internal function FN. */
-static optab
+optab
direct_internal_fn_optab (internal_fn fn, tree_pair types)
{
switch (fn)
@@ -3971,6 +4036,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -4044,6 +4112,68 @@ first_commutative_argument (internal_fn fn)
}
}
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as wide as the element size of the input vectors. */
+
+bool
+widening_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_WIDENING_OPTAB_FN
+ #define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME: \
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ case IFN_##NAME##_EVEN: \
+ case IFN_##NAME##_ODD: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_WIDENING_OPTAB_FN
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as narrow as the element size of the input vectors. */
+
+bool
+narrowing_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_NARROWING_OPTAB_FN
+ #define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, F, O, T) \
+ case IFN_##NAME##: \
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_NARROWING_OPTAB_FN
+
+ default:
+ return false;
+ }
+}
+
/* Return true if IFN_SET_EDOM is supported. */
bool
@@ -4072,6 +4202,8 @@ set_edom_supported_p (void)
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_FN
+#undef DEF_INTERNAL_SIGNED_OPTAB_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
@@ -4080,6 +4212,7 @@ set_edom_supported_p (void)
where STMT is the statement that performs the call. */
static void (*const internal_fn_expanders[]) (internal_fn, gcall *) = {
+
#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) expand_##CODE,
#include "internal-fn.def"
0
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 7fe742c2ae713e7152ab05cfdfba86e4e0aa3456..e9edaa201ad4ad171a49119efa9d6bff49add9f4 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -85,6 +85,34 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_WIDENING_OPTAB_FN is a wrapper that defines five internal
+ functions with DEF_INTERNAL_SIGNED_OPTAB_FN:
+ - one that describes a widening operation with the same number of elements
+ in the output and input vectors,
+ - two that describe a pair of high-low widening operations where the output
+ vectors each have half the number of elements of the input vectors,
+ corresponding to the result of the widening operation on the top half and
+ bottom half, these have the suffixes _HI and _LO,
+ - and two that describe a pair of even-odd widening operations where the
+ output vectors each have half the number of elements of the input vectors,
+ corresponding to the result of the widening operation on the even and odd
+ elements, these have the suffixes _EVEN and _ODD.
+ These five internal functions will require two optabs each, a SIGNED_OPTAB
+ and an UNSIGNED_OTPAB.
+
+ DEF_INTERNAL_NARROWING_OPTAB_FN is a wrapper that defines five internal
+ functions with DEF_INTERNAL_OPTAB_FN:
+ - one that describes a narrowing operation with the same number of elements
+ in the output and input vectors,
+ - two that describe a pair of high-low narrowing operations where the output
+ vector has the same number of elements in the top or bottom halves as the
+ full input vectors, these have the suffixes _HI and _LO.
+ - and two that describe a pair of even-odd narrowing operations where the
+ output vector has the same number of elements, in the even or odd positions,
+ as the full input vectors, these have the suffixes _EVEN and _ODD.
+ These five internal functions will require an optab each.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -123,6 +151,24 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_WIDENING_OPTAB_FN
+#define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, FLAGS, SELECTOR, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME, FLAGS, SELECTOR, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _LO, FLAGS, SELECTOR, SOPTAB##_lo, UOPTAB##_lo, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _HI, FLAGS, SELECTOR, SOPTAB##_hi, UOPTAB##_hi, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _EVEN, FLAGS, SELECTOR, SOPTAB##_even, UOPTAB##_even, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _ODD, FLAGS, SELECTOR, SOPTAB##_odd, UOPTAB##_odd, TYPE)
+#endif
+
+#ifndef DEF_INTERNAL_NARROWING_OPTAB_FN
+#define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, OPTAB##_lo, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, OPTAB##_hi, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _EVEN, FLAGS, OPTAB##_even, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _ODD, FLAGS, OPTAB##_odd, TYPE)
+#endif
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -315,6 +361,16 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_WIDENING_OPTAB_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_sadd, vec_widen_uadd,
+ binary)
+DEF_INTERNAL_WIDENING_OPTAB_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_ssub, vec_widen_usub,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 08922ed4254898f5fffca3f33973e96ed9ce772f..3904ba3ca36949d844532a6a9303f550533311a4 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,10 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern void lookup_hilo_internal_fn (internal_fn, internal_fn *, internal_fn *);
+extern void lookup_evenodd_internal_fn (internal_fn, internal_fn *,
+ internal_fn *);
+extern optab direct_internal_fn_optab (internal_fn, tree_pair);
/* Return the ECF_* flags for function FN. */
@@ -210,6 +218,8 @@ extern bool commutative_binary_fn_p (internal_fn);
extern bool commutative_ternary_fn_p (internal_fn);
extern int first_commutative_argument (internal_fn);
extern bool associative_binary_fn_p (internal_fn);
+extern bool widening_fn_p (code_helper);
+extern bool narrowing_fn_p (code_helper);
extern bool set_edom_supported_p (void);
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index c8e39c82d57a7d726e7da33d247b80f32ec9236c..5a08d91e550b2d92e9572211f811fdba99a33a38 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,15 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_saddl_hi_optab
+ || binoptab == vec_widen_saddl_lo_optab
+ || binoptab == vec_widen_uaddl_hi_optab
+ || binoptab == vec_widen_uaddl_lo_optab
+ || binoptab == vec_widen_sadd_hi_optab
+ || binoptab == vec_widen_sadd_lo_optab
+ || binoptab == vec_widen_uadd_hi_optab
+ || binoptab == vec_widen_uadd_lo_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 695f5911b300c9ca5737de9be809fa01aabe5e01..d41ed6e1afaddd019c7470f965c0ad21c8b2b9d7 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -410,6 +410,16 @@ OPTAB_D (vec_widen_ssubl_hi_optab, "vec_widen_ssubl_hi_$a")
OPTAB_D (vec_widen_ssubl_lo_optab, "vec_widen_ssubl_lo_$a")
OPTAB_D (vec_widen_saddl_hi_optab, "vec_widen_saddl_hi_$a")
OPTAB_D (vec_widen_saddl_lo_optab, "vec_widen_saddl_lo_$a")
+OPTAB_D (vec_widen_ssub_optab, "vec_widen_ssub_$a")
+OPTAB_D (vec_widen_ssub_hi_optab, "vec_widen_ssub_hi_$a")
+OPTAB_D (vec_widen_ssub_lo_optab, "vec_widen_ssub_lo_$a")
+OPTAB_D (vec_widen_ssub_odd_optab, "vec_widen_ssub_odd_$a")
+OPTAB_D (vec_widen_ssub_even_optab, "vec_widen_ssub_even_$a")
+OPTAB_D (vec_widen_sadd_optab, "vec_widen_sadd_$a")
+OPTAB_D (vec_widen_sadd_hi_optab, "vec_widen_sadd_hi_$a")
+OPTAB_D (vec_widen_sadd_lo_optab, "vec_widen_sadd_lo_$a")
+OPTAB_D (vec_widen_sadd_odd_optab, "vec_widen_sadd_odd_$a")
+OPTAB_D (vec_widen_sadd_even_optab, "vec_widen_sadd_even_$a")
OPTAB_D (vec_widen_sshiftl_hi_optab, "vec_widen_sshiftl_hi_$a")
OPTAB_D (vec_widen_sshiftl_lo_optab, "vec_widen_sshiftl_lo_$a")
OPTAB_D (vec_widen_umult_even_optab, "vec_widen_umult_even_$a")
@@ -422,6 +432,16 @@ OPTAB_D (vec_widen_usubl_hi_optab, "vec_widen_usubl_hi_$a")
OPTAB_D (vec_widen_usubl_lo_optab, "vec_widen_usubl_lo_$a")
OPTAB_D (vec_widen_uaddl_hi_optab, "vec_widen_uaddl_hi_$a")
OPTAB_D (vec_widen_uaddl_lo_optab, "vec_widen_uaddl_lo_$a")
+OPTAB_D (vec_widen_usub_optab, "vec_widen_usub_$a")
+OPTAB_D (vec_widen_usub_hi_optab, "vec_widen_usub_hi_$a")
+OPTAB_D (vec_widen_usub_lo_optab, "vec_widen_usub_lo_$a")
+OPTAB_D (vec_widen_usub_odd_optab, "vec_widen_usub_odd_$a")
+OPTAB_D (vec_widen_usub_even_optab, "vec_widen_usub_even_$a")
+OPTAB_D (vec_widen_uadd_optab, "vec_widen_uadd_$a")
+OPTAB_D (vec_widen_uadd_hi_optab, "vec_widen_uadd_hi_$a")
+OPTAB_D (vec_widen_uadd_lo_optab, "vec_widen_uadd_lo_$a")
+OPTAB_D (vec_widen_uadd_odd_optab, "vec_widen_uadd_odd_$a")
+OPTAB_D (vec_widen_uadd_even_optab, "vec_widen_uadd_even_$a")
OPTAB_D (vec_addsub_optab, "vec_addsub$a3")
OPTAB_D (vec_fmaddsub_optab, "vec_fmaddsub$a4")
OPTAB_D (vec_fmsubadd_optab, "vec_fmsubadd$a4")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..7037673d32bd780e1c9b58a51e58e2bac3b30b7e 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..83bc1edb6105f47114b665e24a13e6194b2179a2 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index 0aeebb67fac864db284985f4a6f0653af281d62b..0e847cd04ca6e33f67a86a78a36d35d42aba2627 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -65,6 +65,7 @@ along with GCC; see the file COPYING3. If not see
#include "asan.h"
#include "profile.h"
#include "sreal.h"
+#include "internal-fn.h"
/* This file contains functions for building the Control Flow Graph (CFG)
for a function tree. */
@@ -3411,6 +3412,40 @@ verify_gimple_call (gcall *stmt)
debug_generic_stmt (fn);
return true;
}
+ internal_fn ifn = gimple_call_internal_fn (stmt);
+ if (ifn == IFN_LAST)
+ {
+ error ("gimple call has an invalid IFN");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ else if (widening_fn_p (ifn)
+ || narrowing_fn_p (ifn))
+ {
+ tree lhs = gimple_get_lhs (stmt);
+ if (!lhs)
+ {
+ error ("vector IFN call with no lhs");
+ debug_generic_stmt (fn);
+ return true;
+ }
+
+ bool non_vector_operands = false;
+ for (unsigned i = 0; i < gimple_call_num_args (stmt); ++i)
+ if (!VECTOR_TYPE_P (TREE_TYPE (gimple_call_arg (stmt, i))))
+ {
+ non_vector_operands = true;
+ break;
+ }
+
+ if (non_vector_operands
+ || !VECTOR_TYPE_P (TREE_TYPE (lhs)))
+ {
+ error ("invalid non-vector operands in vector IFN call");
+ debug_generic_stmt (fn);
+ return true;
+ }
+ }
}
else
{
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 63a19f8d1d89c6bd5d8e55a299cbffaa324b4b84..d74d8db2173b1ab117250fea89de5212d5e354ec 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -4433,7 +4433,20 @@ estimate_num_insns (gimple *stmt, eni_weights *weights)
tree decl;
if (gimple_call_internal_p (stmt))
- return 0;
+ {
+ internal_fn fn = gimple_call_internal_fn (stmt);
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS_HI:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_MINUS_HI:
+ case IFN_VEC_WIDEN_MINUS_LO:
+ return 1;
+
+ default:
+ return 0;
+ }
+ }
else if ((decl = gimple_call_fndecl (stmt))
&& fndecl_built_in_p (decl))
{
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 1778af0242898e3dc73d94d22a5b8505628a53b5..dcd4b5561600346a2c10bd5133507329206e8837 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -562,21 +562,30 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
+ return 0;
+
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else if (is_gimple_call (stmt))
+ rhs_code = gimple_call_combined_fn (stmt);
+ else
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ if (rhs_code != code
+ && rhs_code != widened_code)
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = gimple_get_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -589,7 +598,7 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op = gimple_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1343,7 +1352,8 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ IFN_VEC_WIDEN_MINUS,
false, 2, unprom, &half_type))
return NULL;
@@ -1395,14 +1405,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1468,6 +1480,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1481,26 +1507,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_ctz_ffs_pattern
@@ -3078,7 +3108,7 @@ vect_recog_average_pattern (vec_info *vinfo,
vect_unpromoted_value unprom[3];
tree new_type;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
+ IFN_VEC_WIDEN_PLUS, false, 3,
unprom, &new_type);
if (nops == 0)
return NULL;
@@ -6469,6 +6499,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index d152ae9ab10b361b88c0f839d6951c43b954750a..132c0337b7f541bfb114c0a3d2abbeffdad79880 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5038,7 +5038,8 @@ vectorizable_conversion (vec_info *vinfo,
bool widen_arith = (code == WIDEN_PLUS_EXPR
|| code == WIDEN_MINUS_EXPR
|| code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
+ || code == WIDEN_LSHIFT_EXPR
+ || widening_fn_p (code));
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -5088,8 +5089,8 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
-
+ || code == WIDEN_MINUS_EXPR
+ || widening_fn_p (code));
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
gimple_call_arg (stmt, 0);
@@ -12478,26 +12479,69 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
- {
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+
+ vec_mode = TYPE_MODE (vectype);
+ if (widening_fn_p (code))
+ {
+ /* If this is an internal fn then we must check whether the target
+ supports either a low-high split or an even-odd split. */
+ internal_fn ifn = as_internal_fn ((combined_fn) code);
+
+ internal_fn lo, hi, even, odd;
+ lookup_hilo_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = direct_internal_fn_optab (lo, {vectype, vectype});
+ optab2 = direct_internal_fn_optab (hi, {vectype, vectype});
+
+ /* If we don't support low-high, then check for even-odd. */
+ if (!optab1
+ || (icode1 = optab_handler (optab1, vec_mode)) == CODE_FOR_nothing
+ || !optab2
+ || (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
+ {
+ lookup_evenodd_internal_fn (ifn, &even, &odd);
+ *code1 = as_combined_fn (even);
+ *code2 = as_combined_fn (odd);
+ optab1 = direct_internal_fn_optab (even, {vectype, vectype});
+ optab2 = direct_internal_fn_optab (odd, {vectype, vectype});
+ }
+ }
+ else if (code.is_tree_code ())
+ {
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P ((tree_code) code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1, vectype, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ }
+ *code1 = c1;
+ *code2 = c2;
}
if (!optab1 || !optab2)
return false;
- vec_mode = TYPE_MODE (vectype);
if ((icode1 = optab_handler (optab1, vec_mode)) == CODE_FOR_nothing
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- if (code.is_tree_code ())
- {
- *code1 = c1;
- *code2 = c2;
- }
-
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
diff --git a/gcc/tree.def b/gcc/tree.def
index 90ceeec0b512bfa5f983359c0af03cc71de32007..b37b0b35927b92a6536e5c2d9805ffce8319a240 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1374,15 +1374,16 @@ DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr", tcc_expression, 3)
DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr", tcc_binary, 2)
/* Widening sad (sum of absolute differences).
- The first two arguments are of type t1 which should be integer.
- The third argument and the result are of type t2, such that t2 is at least
- twice the size of t1. Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
+ The first two arguments are of type t1 which should be a vector of integers.
+ The third argument and the result are of type t2, such that the size of
+ the elements of t2 is at least twice the size of the elements of t1.
+ Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
equivalent to:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = PLUS_EXPR (tmp2, arg3)
or:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = WIDEN_SUM_EXPR (tmp2, arg3)
*/
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-18 17:15 ` Andre Vieira (lists)
@ 2023-05-22 13:06 ` Richard Biener
2023-06-01 16:27 ` Andre Vieira (lists)
0 siblings, 1 reply; 53+ messages in thread
From: Richard Biener @ 2023-05-22 13:06 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Sandiford, Richard Biener, gcc-patches
On Thu, 18 May 2023, Andre Vieira (lists) wrote:
> How about this?
>
> Not sure about the DEF_INTERNAL documentation I rewrote in internal-fn.def,
> was struggling to word these, so improvements welcome!
The even/odd variant optabs are also commutative_optab_p, so is
the vec_widen_sadd without hi/lo or even/odd.
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
do you really want -all? I think you want -details
+ else if (widening_fn_p (ifn)
+ || narrowing_fn_p (ifn))
+ {
+ tree lhs = gimple_get_lhs (stmt);
+ if (!lhs)
+ {
+ error ("vector IFN call with no lhs");
+ debug_generic_stmt (fn);
that's an error because ...? Maybe we want to verify this
for all ECF_CONST|ECF_NOTHROW (or pure instead of const) internal
function calls, but I wouldn't add any verification as part
of this patch (not special to widening/narrowing fns either).
if (gimple_call_internal_p (stmt))
- return 0;
+ {
+ internal_fn fn = gimple_call_internal_fn (stmt);
+ switch (fn)
+ {
+ case IFN_VEC_WIDEN_PLUS_HI:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_MINUS_HI:
+ case IFN_VEC_WIDEN_MINUS_LO:
+ return 1;
this now looks incomplete. I think that we want instead to
have a default: returning 1 and then special-cases we want
to cost as zero. Not sure which - maybe blame tells why
this was added? I think we can deal with this as followup
(likewise the ranger additions).
Otherwise looks good to me.
Thanks,
Richard.
> gcc/ChangeLog:
>
> 2023-04-25 Andre Vieira <andre.simoesdiasvieira@arm.com>
> Joel Hutton <joel.hutton@arm.com>
> Tamar Christina <tamar.christina@arm.com>
>
> * config/aarch64/aarch64-simd.md (vec_widen_<su>addl_lo_<mode>):
> Rename
> this ...
> (vec_widen_<su>add_lo_<mode>): ... to this.
> (vec_widen_<su>addl_hi_<mode>): Rename this ...
> (vec_widen_<su>add_hi_<mode>): ... to this.
> (vec_widen_<su>subl_lo_<mode>): Rename this ...
> (vec_widen_<su>sub_lo_<mode>): ... to this.
> (vec_widen_<su>subl_hi_<mode>): Rename this ...
> (vec_widen_<su>sub_hi_<mode>): ...to this.
> * doc/generic.texi: Document new IFN codes.
> * internal-fn.cc (ifn_cmp): Function to compare ifn's for
> sorting/searching.
> (lookup_hilo_internal_fn): Add lookup function.
> (commutative_binary_fn_p): Add widen_plus fn's.
> (widening_fn_p): New function.
> (narrowing_fn_p): New function.
> (direct_internal_fn_optab): Change visibility.
> * internal-fn.def (DEF_INTERNAL_WIDENING_OPTAB_FN): Macro to define an
> internal_fn that expands into multiple internal_fns for widening.
> (DEF_INTERNAL_NARROWING_OPTAB_FN): Likewise but for narrowing.
> (IFN_VEC_WIDEN_PLUS, IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO,
> IFN_VEC_WIDEN_PLUS_EVEN, IFN_VEC_WIDEN_PLUS_ODD,
> IFN_VEC_WIDEN_MINUS, IFN_VEC_WIDEN_MINUS_HI,
> IFN_VEC_WIDEN_MINUS_LO,
> IFN_VEC_WIDEN_MINUS_ODD, IFN_VEC_WIDEN_MINUS_EVEN): Define widening
> plus,minus functions.
> * internal-fn.h (direct_internal_fn_optab): Declare new prototype.
> (lookup_hilo_internal_fn): Likewise.
> (widening_fn_p): Likewise.
> (Narrowing_fn_p): Likewise.
> * optabs.cc (commutative_optab_p): Add widening plus optabs.
> * optabs.def (OPTAB_D): Define widen add, sub optabs.
> * tree-cfg.cc (verify_gimple_call): Add checks for widening ifns.
> * tree-inline.cc (estimate_num_insns): Return same
> cost for widen add and sub IFNs as previous tree_codes.
> * tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
> patterns with a hi/lo or even/odd split.
> (vect_recog_sad_pattern): Refactor to use new IFN codes.
> (vect_recog_widen_plus_pattern): Likewise.
> (vect_recog_widen_minus_pattern): Likewise.
> (vect_recog_average_pattern): Likewise.
> * tree-vect-stmts.cc (vectorizable_conversion): Add support for
> _HILO IFNs.
> (supportable_widening_operation): Likewise.
> * tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
>
> gcc/testsuite/ChangeLog:
>
> * gcc.target/aarch64/vect-widen-add.c: Test that new
> IFN_VEC_WIDEN_PLUS is being used.
> * gcc.target/aarch64/vect-widen-sub.c: Test that new
> IFN_VEC_WIDEN_MINUS is being used.
>
--
Richard Biener <rguenther@suse.de>
SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg,
Germany; GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman;
HRB 36809 (AG Nuernberg)
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-05-22 13:06 ` Richard Biener
@ 2023-06-01 16:27 ` Andre Vieira (lists)
2023-06-02 12:00 ` Richard Sandiford
2023-06-06 19:00 ` Jakub Jelinek
0 siblings, 2 replies; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-06-01 16:27 UTC (permalink / raw)
To: Richard Biener; +Cc: Richard Sandiford, Richard Biener, gcc-patches
[-- Attachment #1: Type: text/plain, Size: 8566 bytes --]
Hi,
This is the updated patch and cover letter. Patches for inline and
gimple-op changes will follow soon.
DEF_INTERNAL_WIDENING_OPTAB_FN and DEF_INTERNAL_NARROWING_OPTAB_FN
are like DEF_INTERNAL_SIGNED_OPTAB_FN and DEF_INTERNAL_OPTAB_FN
respectively. With the exception that they provide convenience wrappers
for a single vector to vector conversion, a hi/lo split or an even/odd
split. Each definition for <NAME> will require either signed optabs
named <UOPTAB> and <SOPTAB> (for widening) or a single <OPTAB> (for
narrowing) for each of the five functions it creates.
For example, for widening addition the
DEF_INTERNAL_WIDENING_OPTAB_FN will create five internal functions:
IFN_VEC_WIDEN_PLUS, IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO,
IFN_VEC_WIDEN_PLUS_EVEN and IFN_VEC_WIDEN_PLUS_ODD. Each requiring two
optabs, one for signed and one for unsigned.
Aarch64 implements the hi/lo split optabs:
IFN_VEC_WIDEN_PLUS_HI -> vec_widen_<su>add_hi_<mode> -> (u/s)addl2
IFN_VEC_WIDEN_PLUS_LO -> vec_widen_<su>add_lo_<mode> -> (u/s)addl
This gives the same functionality as the previous
WIDEN_PLUS/WIDEN_MINUS tree codes which are expanded into
VEC_WIDEN_PLUS_LO, VEC_WIDEN_PLUS_HI.
gcc/ChangeLog:
2023-04-25 Andre Vieira <andre.simoesdiasvieira@arm.com>
Joel Hutton <joel.hutton@arm.com>
Tamar Christina <tamar.christina@arm.com>
* config/aarch64/aarch64-simd.md
(vec_widen_<su>addl_lo_<mode>): Rename
this ...
(vec_widen_<su>add_lo_<mode>): ... to this.
(vec_widen_<su>addl_hi_<mode>): Rename this ...
(vec_widen_<su>add_hi_<mode>): ... to this.
(vec_widen_<su>subl_lo_<mode>): Rename this ...
(vec_widen_<su>sub_lo_<mode>): ... to this.
(vec_widen_<su>subl_hi_<mode>): Rename this ...
(vec_widen_<su>sub_hi_<mode>): ...to this.
* doc/generic.texi: Document new IFN codes.
* internal-fn.cc (ifn_cmp): Function to compare ifn's for
sorting/searching.
(lookup_hilo_internal_fn): Add lookup function.
(commutative_binary_fn_p): Add widen_plus fn's.
(widening_fn_p): New function.
(narrowing_fn_p): New function.
(direct_internal_fn_optab): Change visibility.
* internal-fn.def (DEF_INTERNAL_WIDENING_OPTAB_FN): Macro to define an
internal_fn that expands into multiple internal_fns for widening.
(DEF_INTERNAL_NARROWING_OPTAB_FN): Likewise but for narrowing.
(IFN_VEC_WIDEN_PLUS, IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO,
IFN_VEC_WIDEN_PLUS_EVEN, IFN_VEC_WIDEN_PLUS_ODD,
IFN_VEC_WIDEN_MINUS, IFN_VEC_WIDEN_MINUS_HI,
IFN_VEC_WIDEN_MINUS_LO,
IFN_VEC_WIDEN_MINUS_ODD, IFN_VEC_WIDEN_MINUS_EVEN): Define
widening
plus,minus functions.
* internal-fn.h (direct_internal_fn_optab): Declare new prototype.
(lookup_hilo_internal_fn): Likewise.
(widening_fn_p): Likewise.
(Narrowing_fn_p): Likewise.
* optabs.cc (commutative_optab_p): Add widening plus optabs.
* optabs.def (OPTAB_D): Define widen add, sub optabs.
* tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
patterns with a hi/lo or even/odd split.
(vect_recog_sad_pattern): Refactor to use new IFN codes.
(vect_recog_widen_plus_pattern): Likewise.
(vect_recog_widen_minus_pattern): Likewise.
(vect_recog_average_pattern): Likewise.
* tree-vect-stmts.cc (vectorizable_conversion): Add support for
_HILO IFNs.
(supportable_widening_operation): Likewise.
* tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-widen-add.c: Test that new
IFN_VEC_WIDEN_PLUS is being used.
* gcc.target/aarch64/vect-widen-sub.c: Test that new
IFN_VEC_WIDEN_MINUS is being used.
On 22/05/2023 14:06, Richard Biener wrote:
> On Thu, 18 May 2023, Andre Vieira (lists) wrote:
>
>> How about this?
>>
>> Not sure about the DEF_INTERNAL documentation I rewrote in internal-fn.def,
>> was struggling to word these, so improvements welcome!
>
> The even/odd variant optabs are also commutative_optab_p, so is
> the vec_widen_sadd without hi/lo or even/odd.
>
> +/* { dg-options "-O3 -save-temps -fdump-tree-vect-all" } */
>
> do you really want -all? I think you want -details
>
> + else if (widening_fn_p (ifn)
> + || narrowing_fn_p (ifn))
> + {
> + tree lhs = gimple_get_lhs (stmt);
> + if (!lhs)
> + {
> + error ("vector IFN call with no lhs");
> + debug_generic_stmt (fn);
>
> that's an error because ...? Maybe we want to verify this
> for all ECF_CONST|ECF_NOTHROW (or pure instead of const) internal
> function calls, but I wouldn't add any verification as part
> of this patch (not special to widening/narrowing fns either).
>
> if (gimple_call_internal_p (stmt))
> - return 0;
> + {
> + internal_fn fn = gimple_call_internal_fn (stmt);
> + switch (fn)
> + {
> + case IFN_VEC_WIDEN_PLUS_HI:
> + case IFN_VEC_WIDEN_PLUS_LO:
> + case IFN_VEC_WIDEN_MINUS_HI:
> + case IFN_VEC_WIDEN_MINUS_LO:
> + return 1;
>
> this now looks incomplete. I think that we want instead to
> have a default: returning 1 and then special-cases we want
> to cost as zero. Not sure which - maybe blame tells why
> this was added? I think we can deal with this as followup
> (likewise the ranger additions).
>
> Otherwise looks good to me.
>
> Thanks,
> Richard.
>
>> gcc/ChangeLog:
>>
>> 2023-04-25 Andre Vieira <andre.simoesdiasvieira@arm.com>
>> Joel Hutton <joel.hutton@arm.com>
>> Tamar Christina <tamar.christina@arm.com>
>>
>> * config/aarch64/aarch64-simd.md (vec_widen_<su>addl_lo_<mode>):
>> Rename
>> this ...
>> (vec_widen_<su>add_lo_<mode>): ... to this.
>> (vec_widen_<su>addl_hi_<mode>): Rename this ...
>> (vec_widen_<su>add_hi_<mode>): ... to this.
>> (vec_widen_<su>subl_lo_<mode>): Rename this ...
>> (vec_widen_<su>sub_lo_<mode>): ... to this.
>> (vec_widen_<su>subl_hi_<mode>): Rename this ...
>> (vec_widen_<su>sub_hi_<mode>): ...to this.
>> * doc/generic.texi: Document new IFN codes.
>> * internal-fn.cc (ifn_cmp): Function to compare ifn's for
>> sorting/searching.
>> (lookup_hilo_internal_fn): Add lookup function.
>> (commutative_binary_fn_p): Add widen_plus fn's.
>> (widening_fn_p): New function.
>> (narrowing_fn_p): New function.
>> (direct_internal_fn_optab): Change visibility.
>> * internal-fn.def (DEF_INTERNAL_WIDENING_OPTAB_FN): Macro to define an
>> internal_fn that expands into multiple internal_fns for widening.
>> (DEF_INTERNAL_NARROWING_OPTAB_FN): Likewise but for narrowing.
>> (IFN_VEC_WIDEN_PLUS, IFN_VEC_WIDEN_PLUS_HI, IFN_VEC_WIDEN_PLUS_LO,
>> IFN_VEC_WIDEN_PLUS_EVEN, IFN_VEC_WIDEN_PLUS_ODD,
>> IFN_VEC_WIDEN_MINUS, IFN_VEC_WIDEN_MINUS_HI,
>> IFN_VEC_WIDEN_MINUS_LO,
>> IFN_VEC_WIDEN_MINUS_ODD, IFN_VEC_WIDEN_MINUS_EVEN): Define widening
>> plus,minus functions.
>> * internal-fn.h (direct_internal_fn_optab): Declare new prototype.
>> (lookup_hilo_internal_fn): Likewise.
>> (widening_fn_p): Likewise.
>> (Narrowing_fn_p): Likewise.
>> * optabs.cc (commutative_optab_p): Add widening plus optabs.
>> * optabs.def (OPTAB_D): Define widen add, sub optabs.
>> * tree-cfg.cc (verify_gimple_call): Add checks for widening ifns.
>> * tree-inline.cc (estimate_num_insns): Return same
>> cost for widen add and sub IFNs as previous tree_codes.
>> * tree-vect-patterns.cc (vect_recog_widen_op_pattern): Support
>> patterns with a hi/lo or even/odd split.
>> (vect_recog_sad_pattern): Refactor to use new IFN codes.
>> (vect_recog_widen_plus_pattern): Likewise.
>> (vect_recog_widen_minus_pattern): Likewise.
>> (vect_recog_average_pattern): Likewise.
>> * tree-vect-stmts.cc (vectorizable_conversion): Add support for
>> _HILO IFNs.
>> (supportable_widening_operation): Likewise.
>> * tree.def (WIDEN_SUM_EXPR): Update example to use new IFNs.
>>
>> gcc/testsuite/ChangeLog:
>>
>> * gcc.target/aarch64/vect-widen-add.c: Test that new
>> IFN_VEC_WIDEN_PLUS is being used.
>> * gcc.target/aarch64/vect-widen-sub.c: Test that new
>> IFN_VEC_WIDEN_MINUS is being used.
>>
>
[-- Attachment #2: ifn1v5.patch --]
[-- Type: text/plain, Size: 34259 bytes --]
diff --git a/gcc/config/aarch64/aarch64-simd.md b/gcc/config/aarch64/aarch64-simd.md
index da9c59e655465a74926b81b95b4ac8c353efb1b7..b404d5cabf9df8ea8c70ea4537deb978d351c51e 100644
--- a/gcc/config/aarch64/aarch64-simd.md
+++ b/gcc/config/aarch64/aarch64-simd.md
@@ -4626,7 +4626,7 @@
[(set_attr "type" "neon_<ADDSUB:optab>_long")]
)
-(define_expand "vec_widen_<su>addl_lo_<mode>"
+(define_expand "vec_widen_<su>add_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4638,7 +4638,7 @@
DONE;
})
-(define_expand "vec_widen_<su>addl_hi_<mode>"
+(define_expand "vec_widen_<su>add_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4650,7 +4650,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_lo_<mode>"
+(define_expand "vec_widen_<su>sub_lo_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
@@ -4662,7 +4662,7 @@
DONE;
})
-(define_expand "vec_widen_<su>subl_hi_<mode>"
+(define_expand "vec_widen_<su>sub_hi_<mode>"
[(match_operand:<VWIDE> 0 "register_operand")
(ANY_EXTEND:<VWIDE> (match_operand:VQW 1 "register_operand"))
(ANY_EXTEND:<VWIDE> (match_operand:VQW 2 "register_operand"))]
diff --git a/gcc/doc/generic.texi b/gcc/doc/generic.texi
index 8b2882da4fe7da07d22b4e5384d049ba7d3907bf..5e36dac2b1a10257616f12cdfb0b12d0f2879ae9 100644
--- a/gcc/doc/generic.texi
+++ b/gcc/doc/generic.texi
@@ -1811,10 +1811,16 @@ a value from @code{enum annot_expr_kind}, the third is an @code{INTEGER_CST}.
@tindex VEC_RSHIFT_EXPR
@tindex VEC_WIDEN_MULT_HI_EXPR
@tindex VEC_WIDEN_MULT_LO_EXPR
-@tindex VEC_WIDEN_PLUS_HI_EXPR
-@tindex VEC_WIDEN_PLUS_LO_EXPR
-@tindex VEC_WIDEN_MINUS_HI_EXPR
-@tindex VEC_WIDEN_MINUS_LO_EXPR
+@tindex IFN_VEC_WIDEN_PLUS
+@tindex IFN_VEC_WIDEN_PLUS_HI
+@tindex IFN_VEC_WIDEN_PLUS_LO
+@tindex IFN_VEC_WIDEN_PLUS_EVEN
+@tindex IFN_VEC_WIDEN_PLUS_ODD
+@tindex IFN_VEC_WIDEN_MINUS
+@tindex IFN_VEC_WIDEN_MINUS_HI
+@tindex IFN_VEC_WIDEN_MINUS_LO
+@tindex IFN_VEC_WIDEN_MINUS_EVEN
+@tindex IFN_VEC_WIDEN_MINUS_ODD
@tindex VEC_UNPACK_HI_EXPR
@tindex VEC_UNPACK_LO_EXPR
@tindex VEC_UNPACK_FLOAT_HI_EXPR
@@ -1861,6 +1867,82 @@ vector of @code{N/2} products. In the case of @code{VEC_WIDEN_MULT_LO_EXPR} the
low @code{N/2} elements of the two vector are multiplied to produce the
vector of @code{N/2} products.
+@item IFN_VEC_WIDEN_PLUS
+This internal function represents widening vector addition of two input
+vectors. Its operands are vectors that contain the same number of elements
+(@code{N}) of the same integral type. The result is a vector that contains
+the same amount (@code{N}) of elements, of an integral type whose size is twice
+as wide, as the input vectors. If the current target does not implement the
+corresponding optabs the vectorizer may choose to split it into either a pair
+of @code{IFN_VEC_WIDEN_PLUS_HI} and @code{IFN_VEC_WIDEN_PLUS_LO} or
+@code{IFN_VEC_WIDEN_PLUS_EVEN} and @code{IFN_VEC_WIDEN_PLUS_ODD}, depending
+on what optabs the target implements.
+
+@item IFN_VEC_WIDEN_PLUS_HI
+@itemx IFN_VEC_WIDEN_PLUS_LO
+These internal functions represent widening vector addition of the high and low
+parts of the two input vectors, respectively. Their operands are vectors that
+contain the same number of elements (@code{N}) of the same integral type. The
+result is a vector that contains half as many elements, of an integral type
+whose size is twice as wide. In the case of @code{IFN_VEC_WIDEN_PLUS_HI} the
+high @code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions. In the case of @code{IFN_VEC_WIDEN_PLUS_LO} the low
+@code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions.
+
+@item IFN_VEC_WIDEN_PLUS_EVEN
+@itemx IFN_VEC_WIDEN_PLUS_ODD
+These internal functions represent widening vector addition of the even and odd
+elements of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The result is a vector that contains half as many elements, of an integral type
+whose size is twice as wide. In the case of @code{IFN_VEC_WIDEN_PLUS_EVEN} the
+even @code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions. In the case of @code{IFN_VEC_WIDEN_PLUS_ODD} the odd
+@code{N/2} elements of the two vectors are added to produce the vector of
+@code{N/2} additions.
+
+@item IFN_VEC_WIDEN_MINUS
+This internal function represents widening vector subtraction of two input
+vectors. Its operands are vectors that contain the same number of elements
+(@code{N}) of the same integral type. The result is a vector that contains
+the same amount (@code{N}) of elements, of an integral type whose size is twice
+as wide, as the input vectors. If the current target does not implement the
+corresponding optabs the vectorizer may choose to split it into either a pair
+of @code{IFN_VEC_WIDEN_MINUS_HI} and @code{IFN_VEC_WIDEN_MINUS_LO} or
+@code{IFN_VEC_WIDEN_MINUS_EVEN} and @code{IFN_VEC_WIDEN_MINUS_ODD}, depending
+on what optabs the target implements.
+
+@item IFN_VEC_WIDEN_MINUS_HI
+@itemx IFN_VEC_WIDEN_MINUS_LO
+These internal functions represent widening vector subtraction of the high and
+low parts of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The high/low elements of the second vector are subtracted from the high/low
+elements of the first. The result is a vector that contains half as many
+elements, of an integral type whose size is twice as wide. In the case of
+@code{IFN_VEC_WIDEN_MINUS_HI} the high @code{N/2} elements of the second
+vector are subtracted from the high @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions. In the case of
+@code{IFN_VEC_WIDEN_MINUS_LO} the low @code{N/2} elements of the second
+vector are subtracted from the low @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions.
+
+@item IFN_VEC_WIDEN_MINUS_EVEN
+@itemx IFN_VEC_WIDEN_MINUS_ODD
+These internal functions represent widening vector subtraction of the even and
+odd parts of the two input vectors, respectively. Their operands are vectors
+that contain the same number of elements (@code{N}) of the same integral type.
+The even/odd elements of the second vector are subtracted from the even/odd
+elements of the first. The result is a vector that contains half as many
+elements, of an integral type whose size is twice as wide. In the case of
+@code{IFN_VEC_WIDEN_MINUS_EVEN} the even @code{N/2} elements of the second
+vector are subtracted from the even @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions. In the case of
+@code{IFN_VEC_WIDEN_MINUS_ODD} the odd @code{N/2} elements of the second
+vector are subtracted from the odd @code{N/2} of the first to produce the
+vector of @code{N/2} subtractions.
+
@item VEC_WIDEN_PLUS_HI_EXPR
@itemx VEC_WIDEN_PLUS_LO_EXPR
These nodes represent widening vector addition of the high and low parts of
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 5c9da73ea11f8060b18dcf513599c9694fa4f2ad..348bee35a35ae4ed9a8652f5349f430c2733e1cb 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -90,6 +90,71 @@ lookup_internal_fn (const char *name)
return entry ? *entry : IFN_LAST;
}
+/* Given an internal_fn IFN that is either a widening or narrowing function, return its
+ corresponding LO and HI internal_fns. */
+
+extern void
+lookup_hilo_internal_fn (internal_fn ifn, internal_fn *lo, internal_fn *hi)
+{
+ gcc_assert (widening_fn_p (ifn) || narrowing_fn_p (ifn));
+
+ switch (ifn)
+ {
+ default:
+ gcc_unreachable ();
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+#define DEF_INTERNAL_FN(NAME, FLAGS, TYPE)
+#define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME: \
+ *lo = internal_fn (IFN_##NAME##_LO); \
+ *hi = internal_fn (IFN_##NAME##_HI); \
+ break;
+#define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, F, O, T) \
+ case IFN_##NAME: \
+ *lo = internal_fn (IFN_##NAME##_LO); \
+ *hi = internal_fn (IFN_##NAME##_HI); \
+ break;
+#include "internal-fn.def"
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+ }
+}
+
+extern void
+lookup_evenodd_internal_fn (internal_fn ifn, internal_fn *even,
+ internal_fn *odd)
+{
+ gcc_assert (widening_fn_p (ifn) || narrowing_fn_p (ifn));
+
+ switch (ifn)
+ {
+ default:
+ gcc_unreachable ();
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+#define DEF_INTERNAL_FN(NAME, FLAGS, TYPE)
+#define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME: \
+ *even = internal_fn (IFN_##NAME##_EVEN); \
+ *odd = internal_fn (IFN_##NAME##_ODD); \
+ break;
+#define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, F, O, T) \
+ case IFN_##NAME: \
+ *even = internal_fn (IFN_##NAME##_EVEN); \
+ *odd = internal_fn (IFN_##NAME##_ODD); \
+ break;
+#include "internal-fn.def"
+#undef DEF_INTERNAL_FN
+#undef DEF_INTERNAL_WIDENING_OPTAB_FN
+#undef DEF_INTERNAL_NARROWING_OPTAB_FN
+ }
+}
+
+
/* Fnspec of each internal function, indexed by function number. */
const_tree internal_fn_fnspec_array[IFN_LAST + 1];
@@ -3852,7 +3917,7 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types,
/* Return the optab used by internal function FN. */
-static optab
+optab
direct_internal_fn_optab (internal_fn fn, tree_pair types)
{
switch (fn)
@@ -3971,6 +4036,9 @@ commutative_binary_fn_p (internal_fn fn)
case IFN_UBSAN_CHECK_MUL:
case IFN_ADD_OVERFLOW:
case IFN_MUL_OVERFLOW:
+ case IFN_VEC_WIDEN_PLUS:
+ case IFN_VEC_WIDEN_PLUS_LO:
+ case IFN_VEC_WIDEN_PLUS_HI:
return true;
default:
@@ -4044,6 +4112,68 @@ first_commutative_argument (internal_fn fn)
}
}
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as wide as the element size of the input vectors. */
+
+bool
+widening_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_WIDENING_OPTAB_FN
+ #define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, F, S, SO, UO, T) \
+ case IFN_##NAME: \
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ case IFN_##NAME##_EVEN: \
+ case IFN_##NAME##_ODD: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_WIDENING_OPTAB_FN
+
+ default:
+ return false;
+ }
+}
+
+/* Return true if this CODE describes an internal_fn that returns a vector with
+ elements twice as narrow as the element size of the input vectors. */
+
+bool
+narrowing_fn_p (code_helper code)
+{
+ if (!code.is_fn_code ())
+ return false;
+
+ if (!internal_fn_p ((combined_fn) code))
+ return false;
+
+ internal_fn fn = as_internal_fn ((combined_fn) code);
+ switch (fn)
+ {
+ #undef DEF_INTERNAL_NARROWING_OPTAB_FN
+ #define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, F, O, T) \
+ case IFN_##NAME##: \
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ case IFN_##NAME##_HI: \
+ case IFN_##NAME##_LO: \
+ return true;
+ #include "internal-fn.def"
+ #undef DEF_INTERNAL_NARROWING_OPTAB_FN
+
+ default:
+ return false;
+ }
+}
+
/* Return true if IFN_SET_EDOM is supported. */
bool
@@ -4072,6 +4202,8 @@ set_edom_supported_p (void)
expand_##TYPE##_optab_fn (fn, stmt, which_optab); \
}
#include "internal-fn.def"
+#undef DEF_INTERNAL_OPTAB_FN
+#undef DEF_INTERNAL_SIGNED_OPTAB_FN
/* Routines to expand each internal function, indexed by function number.
Each routine has the prototype:
@@ -4080,6 +4212,7 @@ set_edom_supported_p (void)
where STMT is the statement that performs the call. */
static void (*const internal_fn_expanders[]) (internal_fn, gcall *) = {
+
#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) expand_##CODE,
#include "internal-fn.def"
0
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 7fe742c2ae713e7152ab05cfdfba86e4e0aa3456..e9edaa201ad4ad171a49119efa9d6bff49add9f4 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -85,6 +85,34 @@ along with GCC; see the file COPYING3. If not see
says that the function extends the C-level BUILT_IN_<NAME>{,L,LL,IMAX}
group of functions to any integral mode (including vector modes).
+ DEF_INTERNAL_WIDENING_OPTAB_FN is a wrapper that defines five internal
+ functions with DEF_INTERNAL_SIGNED_OPTAB_FN:
+ - one that describes a widening operation with the same number of elements
+ in the output and input vectors,
+ - two that describe a pair of high-low widening operations where the output
+ vectors each have half the number of elements of the input vectors,
+ corresponding to the result of the widening operation on the top half and
+ bottom half, these have the suffixes _HI and _LO,
+ - and two that describe a pair of even-odd widening operations where the
+ output vectors each have half the number of elements of the input vectors,
+ corresponding to the result of the widening operation on the even and odd
+ elements, these have the suffixes _EVEN and _ODD.
+ These five internal functions will require two optabs each, a SIGNED_OPTAB
+ and an UNSIGNED_OTPAB.
+
+ DEF_INTERNAL_NARROWING_OPTAB_FN is a wrapper that defines five internal
+ functions with DEF_INTERNAL_OPTAB_FN:
+ - one that describes a narrowing operation with the same number of elements
+ in the output and input vectors,
+ - two that describe a pair of high-low narrowing operations where the output
+ vector has the same number of elements in the top or bottom halves as the
+ full input vectors, these have the suffixes _HI and _LO.
+ - and two that describe a pair of even-odd narrowing operations where the
+ output vector has the same number of elements, in the even or odd positions,
+ as the full input vectors, these have the suffixes _EVEN and _ODD.
+ These five internal functions will require an optab each.
+
+
Each entry must have a corresponding expander of the form:
void expand_NAME (gimple_call stmt)
@@ -123,6 +151,24 @@ along with GCC; see the file COPYING3. If not see
DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE)
#endif
+#ifndef DEF_INTERNAL_WIDENING_OPTAB_FN
+#define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, FLAGS, SELECTOR, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME, FLAGS, SELECTOR, SOPTAB, UOPTAB, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _LO, FLAGS, SELECTOR, SOPTAB##_lo, UOPTAB##_lo, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _HI, FLAGS, SELECTOR, SOPTAB##_hi, UOPTAB##_hi, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _EVEN, FLAGS, SELECTOR, SOPTAB##_even, UOPTAB##_even, TYPE) \
+ DEF_INTERNAL_SIGNED_OPTAB_FN (NAME ## _ODD, FLAGS, SELECTOR, SOPTAB##_odd, UOPTAB##_odd, TYPE)
+#endif
+
+#ifndef DEF_INTERNAL_NARROWING_OPTAB_FN
+#define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME, FLAGS, OPTAB, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _LO, FLAGS, OPTAB##_lo, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _HI, FLAGS, OPTAB##_hi, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _EVEN, FLAGS, OPTAB##_even, TYPE) \
+ DEF_INTERNAL_OPTAB_FN (NAME ## _ODD, FLAGS, OPTAB##_odd, TYPE)
+#endif
+
DEF_INTERNAL_OPTAB_FN (MASK_LOAD, ECF_PURE, maskload, mask_load)
DEF_INTERNAL_OPTAB_FN (LOAD_LANES, ECF_CONST, vec_load_lanes, load_lanes)
DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE,
@@ -315,6 +361,16 @@ DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, cadd270, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, binary)
DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, cmul_conj, binary)
DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, vec_addsub, binary)
+DEF_INTERNAL_WIDENING_OPTAB_FN (VEC_WIDEN_PLUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_sadd, vec_widen_uadd,
+ binary)
+DEF_INTERNAL_WIDENING_OPTAB_FN (VEC_WIDEN_MINUS,
+ ECF_CONST | ECF_NOTHROW,
+ first,
+ vec_widen_ssub, vec_widen_usub,
+ binary)
DEF_INTERNAL_OPTAB_FN (VEC_FMADDSUB, ECF_CONST, vec_fmaddsub, ternary)
DEF_INTERNAL_OPTAB_FN (VEC_FMSUBADD, ECF_CONST, vec_fmsubadd, ternary)
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 08922ed4254898f5fffca3f33973e96ed9ce772f..3904ba3ca36949d844532a6a9303f550533311a4 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
#ifndef GCC_INTERNAL_FN_H
#define GCC_INTERNAL_FN_H
+#include "insn-codes.h"
+#include "insn-opinit.h"
+
+
/* INTEGER_CST values for IFN_UNIQUE function arg-0.
UNSPEC: Undifferentiated UNIQUE.
@@ -112,6 +116,10 @@ internal_fn_name (enum internal_fn fn)
}
extern internal_fn lookup_internal_fn (const char *);
+extern void lookup_hilo_internal_fn (internal_fn, internal_fn *, internal_fn *);
+extern void lookup_evenodd_internal_fn (internal_fn, internal_fn *,
+ internal_fn *);
+extern optab direct_internal_fn_optab (internal_fn, tree_pair);
/* Return the ECF_* flags for function FN. */
@@ -210,6 +218,8 @@ extern bool commutative_binary_fn_p (internal_fn);
extern bool commutative_ternary_fn_p (internal_fn);
extern int first_commutative_argument (internal_fn);
extern bool associative_binary_fn_p (internal_fn);
+extern bool widening_fn_p (code_helper);
+extern bool narrowing_fn_p (code_helper);
extern bool set_edom_supported_p (void);
diff --git a/gcc/optabs.cc b/gcc/optabs.cc
index a12333c7169fc6219b0e34b6169780f78e033ee3..aab6ab6faf244a8236dac81be2d68fc28819bc9a 100644
--- a/gcc/optabs.cc
+++ b/gcc/optabs.cc
@@ -1314,7 +1314,17 @@ commutative_optab_p (optab binoptab)
|| binoptab == smul_widen_optab
|| binoptab == umul_widen_optab
|| binoptab == smul_highpart_optab
- || binoptab == umul_highpart_optab);
+ || binoptab == umul_highpart_optab
+ || binoptab == vec_widen_sadd_optab
+ || binoptab == vec_widen_uadd_optab
+ || binoptab == vec_widen_sadd_hi_optab
+ || binoptab == vec_widen_sadd_lo_optab
+ || binoptab == vec_widen_uadd_hi_optab
+ || binoptab == vec_widen_uadd_lo_optab
+ || binoptab == vec_widen_sadd_even_optab
+ || binoptab == vec_widen_sadd_odd_optab
+ || binoptab == vec_widen_uadd_even_optab
+ || binoptab == vec_widen_uadd_odd_optab);
}
/* X is to be used in mode MODE as operand OPN to BINOPTAB. If we're
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 695f5911b300c9ca5737de9be809fa01aabe5e01..d41ed6e1afaddd019c7470f965c0ad21c8b2b9d7 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -410,6 +410,16 @@ OPTAB_D (vec_widen_ssubl_hi_optab, "vec_widen_ssubl_hi_$a")
OPTAB_D (vec_widen_ssubl_lo_optab, "vec_widen_ssubl_lo_$a")
OPTAB_D (vec_widen_saddl_hi_optab, "vec_widen_saddl_hi_$a")
OPTAB_D (vec_widen_saddl_lo_optab, "vec_widen_saddl_lo_$a")
+OPTAB_D (vec_widen_ssub_optab, "vec_widen_ssub_$a")
+OPTAB_D (vec_widen_ssub_hi_optab, "vec_widen_ssub_hi_$a")
+OPTAB_D (vec_widen_ssub_lo_optab, "vec_widen_ssub_lo_$a")
+OPTAB_D (vec_widen_ssub_odd_optab, "vec_widen_ssub_odd_$a")
+OPTAB_D (vec_widen_ssub_even_optab, "vec_widen_ssub_even_$a")
+OPTAB_D (vec_widen_sadd_optab, "vec_widen_sadd_$a")
+OPTAB_D (vec_widen_sadd_hi_optab, "vec_widen_sadd_hi_$a")
+OPTAB_D (vec_widen_sadd_lo_optab, "vec_widen_sadd_lo_$a")
+OPTAB_D (vec_widen_sadd_odd_optab, "vec_widen_sadd_odd_$a")
+OPTAB_D (vec_widen_sadd_even_optab, "vec_widen_sadd_even_$a")
OPTAB_D (vec_widen_sshiftl_hi_optab, "vec_widen_sshiftl_hi_$a")
OPTAB_D (vec_widen_sshiftl_lo_optab, "vec_widen_sshiftl_lo_$a")
OPTAB_D (vec_widen_umult_even_optab, "vec_widen_umult_even_$a")
@@ -422,6 +432,16 @@ OPTAB_D (vec_widen_usubl_hi_optab, "vec_widen_usubl_hi_$a")
OPTAB_D (vec_widen_usubl_lo_optab, "vec_widen_usubl_lo_$a")
OPTAB_D (vec_widen_uaddl_hi_optab, "vec_widen_uaddl_hi_$a")
OPTAB_D (vec_widen_uaddl_lo_optab, "vec_widen_uaddl_lo_$a")
+OPTAB_D (vec_widen_usub_optab, "vec_widen_usub_$a")
+OPTAB_D (vec_widen_usub_hi_optab, "vec_widen_usub_hi_$a")
+OPTAB_D (vec_widen_usub_lo_optab, "vec_widen_usub_lo_$a")
+OPTAB_D (vec_widen_usub_odd_optab, "vec_widen_usub_odd_$a")
+OPTAB_D (vec_widen_usub_even_optab, "vec_widen_usub_even_$a")
+OPTAB_D (vec_widen_uadd_optab, "vec_widen_uadd_$a")
+OPTAB_D (vec_widen_uadd_hi_optab, "vec_widen_uadd_hi_$a")
+OPTAB_D (vec_widen_uadd_lo_optab, "vec_widen_uadd_lo_$a")
+OPTAB_D (vec_widen_uadd_odd_optab, "vec_widen_uadd_odd_$a")
+OPTAB_D (vec_widen_uadd_even_optab, "vec_widen_uadd_even_$a")
OPTAB_D (vec_addsub_optab, "vec_addsub$a3")
OPTAB_D (vec_fmaddsub_optab, "vec_fmaddsub$a4")
OPTAB_D (vec_fmsubadd_optab, "vec_fmsubadd$a4")
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
index 220bd9352a4c7acd2e3713e441d74898d3e92b30..b5a73867e44ec3fa04d1201decf81353a67b4c82 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-add.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-details" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_PLUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tuaddl\t} 1} } */
/* { dg-final { scan-assembler-times {\tuaddl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tsaddl\t} 1} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
index a2bed63affbd091977df95a126da1f5b8c1d41d2..1686c3f2f344c367ebb9cf34e558d0878849f9bc 100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-widen-sub.c
@@ -1,5 +1,5 @@
/* { dg-do run } */
-/* { dg-options "-O3 -save-temps" } */
+/* { dg-options "-O3 -save-temps -fdump-tree-vect-details" } */
#include <stdint.h>
#include <string.h>
@@ -86,6 +86,8 @@ main()
return 0;
}
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_LO" "vect" } } */
+/* { dg-final { scan-tree-dump "add new stmt.*VEC_WIDEN_MINUS_HI" "vect" } } */
/* { dg-final { scan-assembler-times {\tusubl\t} 1} } */
/* { dg-final { scan-assembler-times {\tusubl2\t} 1} } */
/* { dg-final { scan-assembler-times {\tssubl\t} 1} } */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 1778af0242898e3dc73d94d22a5b8505628a53b5..dcd4b5561600346a2c10bd5133507329206e8837 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -562,21 +562,30 @@ vect_joust_widened_type (tree type, tree new_type, tree *common_type)
static unsigned int
vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
- tree_code widened_code, bool shift_p,
+ code_helper widened_code, bool shift_p,
unsigned int max_nops,
vect_unpromoted_value *unprom, tree *common_type,
enum optab_subtype *subtype = NULL)
{
/* Check for an integer operation with the right code. */
- gassign *assign = dyn_cast <gassign *> (stmt_info->stmt);
- if (!assign)
+ gimple* stmt = stmt_info->stmt;
+ if (!(is_gimple_assign (stmt) || is_gimple_call (stmt)))
+ return 0;
+
+ code_helper rhs_code;
+ if (is_gimple_assign (stmt))
+ rhs_code = gimple_assign_rhs_code (stmt);
+ else if (is_gimple_call (stmt))
+ rhs_code = gimple_call_combined_fn (stmt);
+ else
return 0;
- tree_code rhs_code = gimple_assign_rhs_code (assign);
- if (rhs_code != code && rhs_code != widened_code)
+ if (rhs_code != code
+ && rhs_code != widened_code)
return 0;
- tree type = TREE_TYPE (gimple_assign_lhs (assign));
+ tree lhs = gimple_get_lhs (stmt);
+ tree type = TREE_TYPE (lhs);
if (!INTEGRAL_TYPE_P (type))
return 0;
@@ -589,7 +598,7 @@ vect_widened_op_tree (vec_info *vinfo, stmt_vec_info stmt_info, tree_code code,
{
vect_unpromoted_value *this_unprom = &unprom[next_op];
unsigned int nops = 1;
- tree op = gimple_op (assign, i + 1);
+ tree op = gimple_arg (stmt, i);
if (i == 1 && TREE_CODE (op) == INTEGER_CST)
{
/* We already have a common type from earlier operands.
@@ -1343,7 +1352,8 @@ vect_recog_sad_pattern (vec_info *vinfo,
/* FORNOW. Can continue analyzing the def-use chain when this stmt in a phi
inside the loop (in case we are analyzing an outer-loop). */
vect_unpromoted_value unprom[2];
- if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR, WIDEN_MINUS_EXPR,
+ if (!vect_widened_op_tree (vinfo, diff_stmt_vinfo, MINUS_EXPR,
+ IFN_VEC_WIDEN_MINUS,
false, 2, unprom, &half_type))
return NULL;
@@ -1395,14 +1405,16 @@ static gimple *
vect_recog_widen_op_pattern (vec_info *vinfo,
stmt_vec_info last_stmt_info, tree *type_out,
tree_code orig_code, code_helper wide_code,
- bool shift_p, const char *name)
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
{
gimple *last_stmt = last_stmt_info->stmt;
vect_unpromoted_value unprom[2];
tree half_type;
if (!vect_widened_op_tree (vinfo, last_stmt_info, orig_code, orig_code,
- shift_p, 2, unprom, &half_type))
+ shift_p, 2, unprom, &half_type, subtype))
+
return NULL;
/* Pattern detected. */
@@ -1468,6 +1480,20 @@ vect_recog_widen_op_pattern (vec_info *vinfo,
type, pattern_stmt, vecctype);
}
+static gimple *
+vect_recog_widen_op_pattern (vec_info *vinfo,
+ stmt_vec_info last_stmt_info, tree *type_out,
+ tree_code orig_code, internal_fn wide_ifn,
+ bool shift_p, const char *name,
+ optab_subtype *subtype = NULL)
+{
+ combined_fn ifn = as_combined_fn (wide_ifn);
+ return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
+ orig_code, ifn, shift_p, name,
+ subtype);
+}
+
+
/* Try to detect multiplication on widened inputs, converting MULT_EXPR
to WIDEN_MULT_EXPR. See vect_recog_widen_op_pattern for details. */
@@ -1481,26 +1507,30 @@ vect_recog_widen_mult_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
}
/* Try to detect addition on widened inputs, converting PLUS_EXPR
- to WIDEN_PLUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_PLUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_plus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- PLUS_EXPR, WIDEN_PLUS_EXPR, false,
- "vect_recog_widen_plus_pattern");
+ PLUS_EXPR, IFN_VEC_WIDEN_PLUS,
+ false, "vect_recog_widen_plus_pattern",
+ &subtype);
}
/* Try to detect subtraction on widened inputs, converting MINUS_EXPR
- to WIDEN_MINUS_EXPR. See vect_recog_widen_op_pattern for details. */
+ to IFN_VEC_WIDEN_MINUS. See vect_recog_widen_op_pattern for details. */
static gimple *
vect_recog_widen_minus_pattern (vec_info *vinfo, stmt_vec_info last_stmt_info,
tree *type_out)
{
+ optab_subtype subtype;
return vect_recog_widen_op_pattern (vinfo, last_stmt_info, type_out,
- MINUS_EXPR, WIDEN_MINUS_EXPR, false,
- "vect_recog_widen_minus_pattern");
+ MINUS_EXPR, IFN_VEC_WIDEN_MINUS,
+ false, "vect_recog_widen_minus_pattern",
+ &subtype);
}
/* Function vect_recog_ctz_ffs_pattern
@@ -3078,7 +3108,7 @@ vect_recog_average_pattern (vec_info *vinfo,
vect_unpromoted_value unprom[3];
tree new_type;
unsigned int nops = vect_widened_op_tree (vinfo, plus_stmt_info, PLUS_EXPR,
- WIDEN_PLUS_EXPR, false, 3,
+ IFN_VEC_WIDEN_PLUS, false, 3,
unprom, &new_type);
if (nops == 0)
return NULL;
@@ -6469,6 +6499,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_mask_conversion_pattern, "mask_conversion" },
{ vect_recog_widen_plus_pattern, "widen_plus" },
{ vect_recog_widen_minus_pattern, "widen_minus" },
+ /* These must come after the double widening ones. */
};
const unsigned int NUM_PATTERNS = ARRAY_SIZE (vect_vect_recog_func_ptrs);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index d73e7f0936435951fe05fa6b787ba053233635aa..4f1569023a4e42ad6d058bccf62687dc3fe1302e 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -5038,7 +5038,8 @@ vectorizable_conversion (vec_info *vinfo,
bool widen_arith = (code == WIDEN_PLUS_EXPR
|| code == WIDEN_MINUS_EXPR
|| code == WIDEN_MULT_EXPR
- || code == WIDEN_LSHIFT_EXPR);
+ || code == WIDEN_LSHIFT_EXPR
+ || widening_fn_p (code));
if (!widen_arith
&& !CONVERT_EXPR_CODE_P (code)
@@ -5088,8 +5089,8 @@ vectorizable_conversion (vec_info *vinfo,
gcc_assert (code == WIDEN_MULT_EXPR
|| code == WIDEN_LSHIFT_EXPR
|| code == WIDEN_PLUS_EXPR
- || code == WIDEN_MINUS_EXPR);
-
+ || code == WIDEN_MINUS_EXPR
+ || widening_fn_p (code));
op1 = is_gimple_assign (stmt) ? gimple_assign_rhs2 (stmt) :
gimple_call_arg (stmt, 0);
@@ -12500,26 +12501,69 @@ supportable_widening_operation (vec_info *vinfo,
optab1 = vec_unpacks_sbool_lo_optab;
optab2 = vec_unpacks_sbool_hi_optab;
}
- else
- {
- optab1 = optab_for_tree_code (c1, vectype, optab_default);
- optab2 = optab_for_tree_code (c2, vectype, optab_default);
+
+ vec_mode = TYPE_MODE (vectype);
+ if (widening_fn_p (code))
+ {
+ /* If this is an internal fn then we must check whether the target
+ supports either a low-high split or an even-odd split. */
+ internal_fn ifn = as_internal_fn ((combined_fn) code);
+
+ internal_fn lo, hi, even, odd;
+ lookup_hilo_internal_fn (ifn, &lo, &hi);
+ *code1 = as_combined_fn (lo);
+ *code2 = as_combined_fn (hi);
+ optab1 = direct_internal_fn_optab (lo, {vectype, vectype});
+ optab2 = direct_internal_fn_optab (hi, {vectype, vectype});
+
+ /* If we don't support low-high, then check for even-odd. */
+ if (!optab1
+ || (icode1 = optab_handler (optab1, vec_mode)) == CODE_FOR_nothing
+ || !optab2
+ || (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
+ {
+ lookup_evenodd_internal_fn (ifn, &even, &odd);
+ *code1 = as_combined_fn (even);
+ *code2 = as_combined_fn (odd);
+ optab1 = direct_internal_fn_optab (even, {vectype, vectype});
+ optab2 = direct_internal_fn_optab (odd, {vectype, vectype});
+ }
+ }
+ else if (code.is_tree_code ())
+ {
+ if (code == FIX_TRUNC_EXPR)
+ {
+ /* The signedness is determined from output operand. */
+ optab1 = optab_for_tree_code (c1, vectype_out, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype_out, optab_default);
+ }
+ else if (CONVERT_EXPR_CODE_P ((tree_code) code.safe_as_tree_code ())
+ && VECTOR_BOOLEAN_TYPE_P (wide_vectype)
+ && VECTOR_BOOLEAN_TYPE_P (vectype)
+ && TYPE_MODE (wide_vectype) == TYPE_MODE (vectype)
+ && SCALAR_INT_MODE_P (TYPE_MODE (vectype)))
+ {
+ /* If the input and result modes are the same, a different optab
+ is needed where we pass in the number of units in vectype. */
+ optab1 = vec_unpacks_sbool_lo_optab;
+ optab2 = vec_unpacks_sbool_hi_optab;
+ }
+ else
+ {
+ optab1 = optab_for_tree_code (c1, vectype, optab_default);
+ optab2 = optab_for_tree_code (c2, vectype, optab_default);
+ }
+ *code1 = c1;
+ *code2 = c2;
}
if (!optab1 || !optab2)
return false;
- vec_mode = TYPE_MODE (vectype);
if ((icode1 = optab_handler (optab1, vec_mode)) == CODE_FOR_nothing
|| (icode2 = optab_handler (optab2, vec_mode)) == CODE_FOR_nothing)
return false;
- if (code.is_tree_code ())
- {
- *code1 = c1;
- *code2 = c2;
- }
-
if (insn_data[icode1].operand[0].mode == TYPE_MODE (wide_vectype)
&& insn_data[icode2].operand[0].mode == TYPE_MODE (wide_vectype))
diff --git a/gcc/tree.def b/gcc/tree.def
index 90ceeec0b512bfa5f983359c0af03cc71de32007..b37b0b35927b92a6536e5c2d9805ffce8319a240 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1374,15 +1374,16 @@ DEFTREECODE (DOT_PROD_EXPR, "dot_prod_expr", tcc_expression, 3)
DEFTREECODE (WIDEN_SUM_EXPR, "widen_sum_expr", tcc_binary, 2)
/* Widening sad (sum of absolute differences).
- The first two arguments are of type t1 which should be integer.
- The third argument and the result are of type t2, such that t2 is at least
- twice the size of t1. Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
+ The first two arguments are of type t1 which should be a vector of integers.
+ The third argument and the result are of type t2, such that the size of
+ the elements of t2 is at least twice the size of the elements of t1.
+ Like DOT_PROD_EXPR, SAD_EXPR (arg1,arg2,arg3) is
equivalent to:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = PLUS_EXPR (tmp2, arg3)
or:
- tmp = WIDEN_MINUS_EXPR (arg1, arg2)
+ tmp = IFN_VEC_WIDEN_MINUS_EXPR (arg1, arg2)
tmp2 = ABS_EXPR (tmp)
arg3 = WIDEN_SUM_EXPR (tmp2, arg3)
*/
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-06-01 16:27 ` Andre Vieira (lists)
@ 2023-06-02 12:00 ` Richard Sandiford
2023-06-06 19:00 ` Jakub Jelinek
1 sibling, 0 replies; 53+ messages in thread
From: Richard Sandiford @ 2023-06-02 12:00 UTC (permalink / raw)
To: Andre Vieira (lists); +Cc: Richard Biener, Richard Biener, gcc-patches
Just some very minor things.
"Andre Vieira (lists)" <andre.simoesdiasvieira@arm.com> writes:
> diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
> index 5c9da73ea11f8060b18dcf513599c9694fa4f2ad..348bee35a35ae4ed9a8652f5349f430c2733e1cb 100644
> --- a/gcc/internal-fn.cc
> +++ b/gcc/internal-fn.cc
> @@ -90,6 +90,71 @@ lookup_internal_fn (const char *name)
> return entry ? *entry : IFN_LAST;
> }
>
> +/* Given an internal_fn IFN that is either a widening or narrowing function, return its
> + corresponding LO and HI internal_fns. */
Long line and too much space after "/*":
/* Given an internal_fn IFN that is either a widening or narrowing function,
return its corresponding _LO and _HI internal_fns in *LO and *HI. */
> +extern void
> +lookup_hilo_internal_fn (internal_fn ifn, internal_fn *lo, internal_fn *hi)
> +{
> + gcc_assert (widening_fn_p (ifn) || narrowing_fn_p (ifn));
> +
> + switch (ifn)
> + {
> + default:
> + gcc_unreachable ();
> +#undef DEF_INTERNAL_FN
> +#undef DEF_INTERNAL_WIDENING_OPTAB_FN
> +#undef DEF_INTERNAL_NARROWING_OPTAB_FN
> +#define DEF_INTERNAL_FN(NAME, FLAGS, TYPE)
> +#define DEF_INTERNAL_WIDENING_OPTAB_FN(NAME, F, S, SO, UO, T) \
> + case IFN_##NAME: \
> + *lo = internal_fn (IFN_##NAME##_LO); \
> + *hi = internal_fn (IFN_##NAME##_HI); \
> + break;
> +#define DEF_INTERNAL_NARROWING_OPTAB_FN(NAME, F, O, T) \
> + case IFN_##NAME: \
> + *lo = internal_fn (IFN_##NAME##_LO); \
> + *hi = internal_fn (IFN_##NAME##_HI); \
> + break;
> +#include "internal-fn.def"
> +#undef DEF_INTERNAL_FN
> +#undef DEF_INTERNAL_WIDENING_OPTAB_FN
> +#undef DEF_INTERNAL_NARROWING_OPTAB_FN
> + }
> +}
> +
> +extern void
> +lookup_evenodd_internal_fn (internal_fn ifn, internal_fn *even,
> + internal_fn *odd)
This needs a similar comment:
/* Given an internal_fn IFN that is either a widening or narrowing function,
return its corresponding _EVEN and _ODD internal_fns in *EVEN and *ODD. */
> @@ -3971,6 +4036,9 @@ commutative_binary_fn_p (internal_fn fn)
> case IFN_UBSAN_CHECK_MUL:
> case IFN_ADD_OVERFLOW:
> case IFN_MUL_OVERFLOW:
> + case IFN_VEC_WIDEN_PLUS:
> + case IFN_VEC_WIDEN_PLUS_LO:
> + case IFN_VEC_WIDEN_PLUS_HI:
Should include even & odd as well.
I'd suggest leaving out the narrowing stuff for now. There are some
questions that would be easier to answer once we add the first use,
such as whether one of the hi/lo pair and one or the even/odd pair
merge with a vector containing the other half, whether all four
define the other half to be zero, etc.
OK for the optab/internal-fn parts with those changes from my POV.
Thanks again for doing this!
Richard
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH 2/3] Refactor widen_plus as internal_fn
2023-06-01 16:27 ` Andre Vieira (lists)
2023-06-02 12:00 ` Richard Sandiford
@ 2023-06-06 19:00 ` Jakub Jelinek
2023-06-06 21:28 ` [PATCH] modula2: Fix bootstrap Jakub Jelinek
1 sibling, 1 reply; 53+ messages in thread
From: Jakub Jelinek @ 2023-06-06 19:00 UTC (permalink / raw)
To: Andre Vieira (lists)
Cc: Richard Biener, Richard Sandiford, Richard Biener, gcc-patches
On Thu, Jun 01, 2023 at 05:27:56PM +0100, Andre Vieira (lists) via Gcc-patches wrote:
> --- a/gcc/internal-fn.h
> +++ b/gcc/internal-fn.h
> @@ -20,6 +20,10 @@ along with GCC; see the file COPYING3. If not see
> #ifndef GCC_INTERNAL_FN_H
> #define GCC_INTERNAL_FN_H
>
> +#include "insn-codes.h"
> +#include "insn-opinit.h"
My i686-linux build configured with
../configure --enable-languages=default,obj-c++,lto,go,d,rust,m2 --enable-checking=yes,rtl,extra --enable-libstdcxx-backtrace=yes
just died with
In file included from ../../gcc/m2/gm2-gcc/gcc-consolidation.h:74,
from ../../gcc/m2/gm2-gcc/m2except.cc:22:
../../gcc/internal-fn.h:24:10: fatal error: insn-opinit.h: No such file or directory
24 | #include "insn-opinit.h"
| ^~~~~~~~~~~~~~~
compilation terminated.
In file included from ../../gcc/m2/gm2-gcc/gcc-consolidation.h:74,
from ../../gcc/m2/m2pp.cc:23:
../../gcc/internal-fn.h:24:10: fatal error: insn-opinit.h: No such file or directory
24 | #include "insn-opinit.h"
| ^~~~~~~~~~~~~~~
In file included from ../../gcc/m2/gm2-gcc/gcc-consolidation.h:74,
from ../../gcc/m2/gm2-gcc/rtegraph.cc:22:
../../gcc/internal-fn.h:24:10: fatal error: insn-opinit.h: No such file or directory
24 | #include "insn-opinit.h"
| ^~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
supposedly because of this change.
Do you really need those includes there?
If yes, what is supposed to ensure that the generated includes
are generated before compiling files which include those?
From what I can see, gcc/Makefile.in has
generated_files var which includes among other things insn-opinit.h,
and
# Dependency information.
# In order for parallel make to really start compiling the expensive
# objects from $(OBJS) as early as possible, build all their
# prerequisites strictly before all objects.
$(ALL_HOST_OBJS) : | $(generated_files)
rule, plus I see $(generated_files) mentioned in a couple of dependencies
in gcc/m2/Make-lang.in . But supposedly because of this change it now
needs to be added to tons of other spots.
Jakub
^ permalink raw reply [flat|nested] 53+ messages in thread
* [PATCH] modula2: Fix bootstrap
2023-06-06 19:00 ` Jakub Jelinek
@ 2023-06-06 21:28 ` Jakub Jelinek
2023-06-06 22:18 ` Gaius Mulley
2023-06-07 8:42 ` Andre Vieira (lists)
0 siblings, 2 replies; 53+ messages in thread
From: Jakub Jelinek @ 2023-06-06 21:28 UTC (permalink / raw)
To: Gaius Mulley; +Cc: Andre Vieira, Richard Biener, Richard Sandiford, gcc-patches
Hi!
internal-fn.h since yesterday includes insn-opinit.h, which is a generated
header.
One of my bootstraps today failed because some m2 sources started compiling
before insn-opinit.h has been generated.
Normally, gcc/Makefile.in has
# In order for parallel make to really start compiling the expensive
# objects from $(OBJS) as early as possible, build all their
# prerequisites strictly before all objects.
$(ALL_HOST_OBJS) : | $(generated_files)
rule which ensures that all the generated files are generated before
any $(ALL_HOST_OBJS) objects start, but use order-only dependency for
this because we don't want to rebuild most of the objects whenever one
generated header is regenerated. After the initial build in an empty
directory we'll have .deps/ files contain the detailed dependencies.
$(ALL_HOST_OBJS) includes even some FE files, I think in the m2 case
would be m2_OBJS, but m2/Make-lang.in doesn't define those.
The following patch just adds a similar rule to m2/Make-lang.in.
Another option would be to set m2_OBJS variable in m2/Make-lang.in to
something, but not really sure to which exactly and why it isn't
done.
Bootstrapped/regtested on x86_64-linux and i686-linux, ok for trunk?
2023-06-06 Jakub Jelinek <jakub@redhat.com>
* Make-lang.in: Build $(generated_files) before building
all $(GM2_C_OBJS).
--- gcc/m2/Make-lang.in.jj 2023-05-04 09:31:27.289948109 +0200
+++ gcc/m2/Make-lang.in 2023-06-06 21:38:26.655336041 +0200
@@ -511,6 +511,8 @@ GM2_LIBS_BOOT = m2/gm2-compiler-boot
m2/gm2-libs-boot/libgm2.a \
$(GM2-BOOT-O)
+$(GM2_C_OBJS) : | $(generated_files)
+
cc1gm2$(exeext): m2/stage1/cc1gm2$(exeext) $(m2.prev)
cp -p $< $@
Jakub
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH] modula2: Fix bootstrap
2023-06-06 21:28 ` [PATCH] modula2: Fix bootstrap Jakub Jelinek
@ 2023-06-06 22:18 ` Gaius Mulley
2023-06-07 8:42 ` Andre Vieira (lists)
1 sibling, 0 replies; 53+ messages in thread
From: Gaius Mulley @ 2023-06-06 22:18 UTC (permalink / raw)
To: Jakub Jelinek
Cc: Andre Vieira, Richard Biener, Richard Sandiford, gcc-patches
Jakub Jelinek <jakub@redhat.com> writes:
> Hi!
>
> internal-fn.h since yesterday includes insn-opinit.h, which is a generated
> header.
> One of my bootstraps today failed because some m2 sources started compiling
> before insn-opinit.h has been generated.
>
> Normally, gcc/Makefile.in has
> # In order for parallel make to really start compiling the expensive
> # objects from $(OBJS) as early as possible, build all their
> # prerequisites strictly before all objects.
> $(ALL_HOST_OBJS) : | $(generated_files)
>
> rule which ensures that all the generated files are generated before
> any $(ALL_HOST_OBJS) objects start, but use order-only dependency for
> this because we don't want to rebuild most of the objects whenever one
> generated header is regenerated. After the initial build in an empty
> directory we'll have .deps/ files contain the detailed dependencies.
>
> $(ALL_HOST_OBJS) includes even some FE files, I think in the m2 case
> would be m2_OBJS, but m2/Make-lang.in doesn't define those.
>
> The following patch just adds a similar rule to m2/Make-lang.in.
> Another option would be to set m2_OBJS variable in m2/Make-lang.in to
> something, but not really sure to which exactly and why it isn't
> done.
>
> Bootstrapped/regtested on x86_64-linux and i686-linux, ok for trunk?
> 2023-06-06 Jakub Jelinek <jakub@redhat.com>
>
> * Make-lang.in: Build $(generated_files) before building
> all $(GM2_C_OBJS).
>
> --- gcc/m2/Make-lang.in.jj 2023-05-04 09:31:27.289948109 +0200
> +++ gcc/m2/Make-lang.in 2023-06-06 21:38:26.655336041 +0200
> @@ -511,6 +511,8 @@ GM2_LIBS_BOOT = m2/gm2-compiler-boot
> m2/gm2-libs-boot/libgm2.a \
> $(GM2-BOOT-O)
>
> +$(GM2_C_OBJS) : | $(generated_files)
> +
> cc1gm2$(exeext): m2/stage1/cc1gm2$(exeext) $(m2.prev)
> cp -p $< $@
>
>
>
> Jakub
Hi Jakub,
sure looks good to me - thanks for the patch,
regards,
Gaius
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH] modula2: Fix bootstrap
2023-06-06 21:28 ` [PATCH] modula2: Fix bootstrap Jakub Jelinek
2023-06-06 22:18 ` Gaius Mulley
@ 2023-06-07 8:42 ` Andre Vieira (lists)
2023-06-13 14:48 ` Jakub Jelinek
1 sibling, 1 reply; 53+ messages in thread
From: Andre Vieira (lists) @ 2023-06-07 8:42 UTC (permalink / raw)
To: Jakub Jelinek, Gaius Mulley
Cc: Richard Biener, Richard Sandiford, gcc-patches
Thanks Jakub!
I do need those includes and sorry I broke your bootstrap it didn't show
up on my aarch64-unknown-linux-gnu bootstrap, I'm guessing the rules
there were just run in a different order. Glad you were able to fix it :)
On 06/06/2023 22:28, Jakub Jelinek wrote:
> Hi!
>
> internal-fn.h since yesterday includes insn-opinit.h, which is a generated
> header.
> One of my bootstraps today failed because some m2 sources started compiling
> before insn-opinit.h has been generated.
>
> Normally, gcc/Makefile.in has
> # In order for parallel make to really start compiling the expensive
> # objects from $(OBJS) as early as possible, build all their
> # prerequisites strictly before all objects.
> $(ALL_HOST_OBJS) : | $(generated_files)
>
> rule which ensures that all the generated files are generated before
> any $(ALL_HOST_OBJS) objects start, but use order-only dependency for
> this because we don't want to rebuild most of the objects whenever one
> generated header is regenerated. After the initial build in an empty
> directory we'll have .deps/ files contain the detailed dependencies.
>
> $(ALL_HOST_OBJS) includes even some FE files, I think in the m2 case
> would be m2_OBJS, but m2/Make-lang.in doesn't define those.
>
> The following patch just adds a similar rule to m2/Make-lang.in.
> Another option would be to set m2_OBJS variable in m2/Make-lang.in to
> something, but not really sure to which exactly and why it isn't
> done.
>
> Bootstrapped/regtested on x86_64-linux and i686-linux, ok for trunk?
>
> 2023-06-06 Jakub Jelinek <jakub@redhat.com>
>
> * Make-lang.in: Build $(generated_files) before building
> all $(GM2_C_OBJS).
>
> --- gcc/m2/Make-lang.in.jj 2023-05-04 09:31:27.289948109 +0200
> +++ gcc/m2/Make-lang.in 2023-06-06 21:38:26.655336041 +0200
> @@ -511,6 +511,8 @@ GM2_LIBS_BOOT = m2/gm2-compiler-boot
> m2/gm2-libs-boot/libgm2.a \
> $(GM2-BOOT-O)
>
> +$(GM2_C_OBJS) : | $(generated_files)
> +
> cc1gm2$(exeext): m2/stage1/cc1gm2$(exeext) $(m2.prev)
> cp -p $< $@
>
>
>
> Jakub
>
^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [PATCH] modula2: Fix bootstrap
2023-06-07 8:42 ` Andre Vieira (lists)
@ 2023-06-13 14:48 ` Jakub Jelinek
0 siblings, 0 replies; 53+ messages in thread
From: Jakub Jelinek @ 2023-06-13 14:48 UTC (permalink / raw)
To: Andre Vieira (lists)
Cc: Gaius Mulley, Richard Biener, Richard Sandiford, gcc-patches
On Wed, Jun 07, 2023 at 09:42:22AM +0100, Andre Vieira (lists) wrote:
> I do need those includes and sorry I broke your bootstrap it didn't show up
> on my aarch64-unknown-linux-gnu bootstrap, I'm guessing the rules there were
> just run in a different order. Glad you were able to fix it :)
Unfortunately, it doesn't really work.
My x86_64-linux bootstrap today died again with:
In file included from ../../gcc/m2/gm2-gcc/gcc-consolidation.h:74,
from ../../gcc/m2/gm2-lang.cc:24:
../../gcc/internal-fn.h:24:10: fatal error: insn-opinit.h: No such file or directory
24 | #include "insn-opinit.h"
| ^~~~~~~~~~~~~~~
compilation terminated.
/home/jakub/src/gcc/obj36/./prev-gcc/xg++ -B/home/jakub/src/gcc/obj36/./prev-gcc/ -B/usr/local/x86_64-pc-linux-gnu/bin/ -nostdinc++ -B/home/jakub/src/gcc/obj36/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -B/home/jakub/src/gcc/obj36/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -I/home/jakub/src/gcc/obj36/prev-x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu -I/home/jakub/src/gcc/obj36/prev-x86_64-pc-linux-gnu/libstdc++-v3/include -I/home/jakub/src/gcc/libstdc++-v3/libsupc++ -L/home/jakub/src/gcc/obj36/prev-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs -L/home/jakub/src/gcc/obj36/prev-x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/.libs -fno-PIE -c -g -g -O2 -fchecking=1 -DIN_GCC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -fno-common -DHAVE_CONFIG_H \
-I. -Im2/gm2-gcc -I../../gcc -I../../gcc/m2/gm2-gcc -I../../gcc/../include -I../../gcc/../libcpp/include -I../../gcc/../libcody -I../../gcc/../libdecnumber -I../../gcc/../libdecnumber/bid -I../libdecnumber -I../../gcc/../libbacktrace -I. -Im2/gm2-gcc -I../../gcc -I../../gcc/m2/gm2-gcc -I../../gcc/../include -I../../gcc/../libcpp/include -I../../gcc/../libcody -I../../gcc/../libdecnumber -I../../gcc/../libdecnumber/bid -I../libdecnumber -I../../gcc/../libbacktrace ../../gcc/m2/gm2-gcc/m2type.cc -o m2/gm2-gcc/m2type.o
make[3]: *** [../../gcc/m2/Make-lang.in:570: m2/gm2-lang.o] Error 1
make[3]: *** Waiting for unfinished jobs....
errors. Dunno what is going on.
I've tried
--- gcc/m2/Make-lang.in.jj 2023-06-07 15:56:07.112684198 +0200
+++ gcc/m2/Make-lang.in 2023-06-13 16:08:55.409364765 +0200
@@ -511,7 +511,7 @@ GM2_LIBS_BOOT = m2/gm2-compiler-boot
m2/gm2-libs-boot/libgm2.a \
$(GM2-BOOT-O)
-$(GM2_C_OBJS) : | $(generated_files)
+m2_OBJS = $(GM2_C_OBJS)
cc1gm2$(exeext): m2/stage1/cc1gm2$(exeext) $(m2.prev)
cp -p $< $@
but that doesn't really work either, this time not just random bootstrap
breakages from time to time, but all the time.
Including GM2_C_OBJS in m2_OBJS is I think the right thing, but that
results in predefining IN_GCC_FRONTEND macro and we have e.g.
/* Front ends should never have to include middle-end headers. Enforce
this by poisoning the header double-include protection defines. */
#ifdef IN_GCC_FRONTEND
#pragma GCC poison GCC_RTL_H GCC_EXCEPT_H GCC_EXPR_H
#endif
in system.h to make sure that FE sources don't include rtl.h, except.h,
expr.h. But m2/gm2-gcc/gcc-consolidation.h includes tons of the RTL
headers, rtl.h, df.h (twice), except.h; why?
Also, seems one of GM2_C_OBJS is some special copy of stor-layout.cc
which really isn't a FE file and so needs the RTL-ish headers.
Jakub
^ permalink raw reply [flat|nested] 53+ messages in thread
end of thread, other threads:[~2023-06-13 14:48 UTC | newest]
Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-25 9:11 [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns Joel Hutton
2022-05-27 13:23 ` Richard Biener
2022-05-31 10:07 ` Joel Hutton
2022-05-31 16:46 ` Tamar Christina
2022-06-01 10:11 ` Richard Biener
2022-06-06 17:20 ` Joel Hutton
2022-06-07 8:18 ` Richard Sandiford
2022-06-07 9:01 ` Joel Hutton
2022-06-09 14:03 ` Joel Hutton
2022-06-13 9:02 ` Richard Biener
2022-06-30 13:20 ` Joel Hutton
2022-07-12 12:32 ` Richard Biener
2023-03-17 10:14 ` Andre Vieira (lists)
2023-03-17 11:52 ` Richard Biener
2023-04-20 13:23 ` Andre Vieira (lists)
2023-04-24 11:57 ` Richard Biener
2023-04-24 13:01 ` Richard Sandiford
2023-04-25 12:30 ` Richard Biener
2023-04-28 16:06 ` Andre Vieira (lists)
2023-04-25 9:55 ` Andre Vieira (lists)
2023-04-28 12:36 ` [PATCH 1/3] Refactor to allow internal_fn's Andre Vieira (lists)
2023-05-03 11:55 ` Richard Biener
2023-05-04 15:20 ` Andre Vieira (lists)
2023-05-05 6:09 ` Richard Biener
2023-05-12 12:14 ` Andre Vieira (lists)
2023-05-12 13:18 ` Richard Biener
2023-04-28 12:37 ` [PATCH 2/3] Refactor widen_plus as internal_fn Andre Vieira (lists)
2023-05-03 12:11 ` Richard Biener
2023-05-03 19:07 ` Richard Sandiford
2023-05-12 12:16 ` Andre Vieira (lists)
2023-05-12 13:28 ` Richard Biener
2023-05-12 13:55 ` Andre Vieira (lists)
2023-05-12 14:01 ` Richard Sandiford
2023-05-15 10:20 ` Richard Biener
2023-05-15 10:47 ` Richard Sandiford
2023-05-15 11:01 ` Richard Biener
2023-05-15 11:10 ` Richard Sandiford
2023-05-15 11:53 ` Andre Vieira (lists)
2023-05-15 12:21 ` Richard Biener
2023-05-18 17:15 ` Andre Vieira (lists)
2023-05-22 13:06 ` Richard Biener
2023-06-01 16:27 ` Andre Vieira (lists)
2023-06-02 12:00 ` Richard Sandiford
2023-06-06 19:00 ` Jakub Jelinek
2023-06-06 21:28 ` [PATCH] modula2: Fix bootstrap Jakub Jelinek
2023-06-06 22:18 ` Gaius Mulley
2023-06-07 8:42 ` Andre Vieira (lists)
2023-06-13 14:48 ` Jakub Jelinek
2023-04-28 12:37 ` [PATCH 3/3] Remove widen_plus/minus_expr tree codes Andre Vieira (lists)
2023-05-03 12:29 ` Richard Biener
2023-05-10 9:15 ` Andre Vieira (lists)
2023-05-12 12:18 ` Andre Vieira (lists)
2022-06-13 9:18 ` [ping][vect-patterns] Refactor widen_plus/widen_minus as internal_fns Richard Biener
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).