From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) by sourceware.org (Postfix) with ESMTPS id F0ACD3853D36 for ; Fri, 18 Aug 2023 09:33:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org F0ACD3853D36 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2bba5d0c687so8268291fa.1 for ; Fri, 18 Aug 2023 02:33:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692351197; x=1692955997; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=CT6By9/jWALCqtF5/OfJxgzWCGFyPfRxNPshcsmpF5o=; b=YaDqeqPiVjxN2RXH03ydjxJoZA/Be2GJ0sqUhjBgbeDg6ActWQwiK1PPngXa1Jtt9n 0gyIrLKF/XjxvyR1AUzV+NLAtfCtm5jK7QVHcrIwx5D2m/FG19Ik+V08zVT9EBz6ruUl daWfFJgDTxCRwGvgRyXa2y4OBXRuTpZQ91Etoz2RtfKtjnyRUuNBwT7HF4c0vPkODXVB L4e2k4LPqxVuZJhgFYpAEGf18LqQbKTrU0UGtB8EZ5YiLVPCSHoWPhWLK9+YM3gzRWpV 7Ax192UwuhbtSSwdiR3SiOxMbYAlFLB4uaoXiu+lMYvNsmb0u+xUE3W6ILvX5wl5glj7 EtgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692351197; x=1692955997; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CT6By9/jWALCqtF5/OfJxgzWCGFyPfRxNPshcsmpF5o=; b=GbzdSWb7TBtzZWqMuijAVQ6DZ+fjBstiQeBe10kvRXJ5glSJUICWVffMLlJg+c2G2k U6GVB+R13c62h2ueOdFHOmTtONAf6RzEVLuIvut3swSo+345jOvR7ot5dcNPttIHVvga cRTvAlVuLKveLimcMnTzd9+IyNSoDO1obmiSoA6rNI3u00buI5R0IMslQWhoSdzb3ZhR FhDsto8VE6Js/PtYsuZxTiJFnI/Ibt5pUj1/gtfdOC/ZfK4JVpnvtSrT1/i8ckesz6Ls A2blSSsevgVea6isAVbEcPAdn8RZM1XmLqahpeefOj6IEO7NoZSWNZZ/RwTUjkwrKHas yqnw== X-Gm-Message-State: AOJu0Yz4ua5AokKXXGr3a5FPU6X7B9SyluwJmLWNbwg3Tlbz/0PeSYHr cX0FaM3BDDQGK/vqO4AhfPeLNW77zrhjfCnmQCt2nJaL X-Google-Smtp-Source: AGHT+IFm8PrQTmDF3UuOt4ufJ+wC4dUyKCCcisC7vlZZZEgDPnKmHWsilXQD0JJfNNJIG0+SG6LtP2fz3EHVWySwNR0= X-Received: by 2002:a2e:7e0c:0:b0:2b8:3a1e:eebd with SMTP id z12-20020a2e7e0c000000b002b83a1eeebdmr638497ljc.21.1692351197026; Fri, 18 Aug 2023 02:33:17 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Richard Biener Date: Fri, 18 Aug 2023 11:33:05 +0200 Message-ID: Subject: Re: [PATCH] vect: Factor out the handling on scatter store having gs_info.decl To: "Kewen.Lin" Cc: GCC Patches , Richard Sandiford Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,GIT_PATCH_0,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: On Thu, Aug 17, 2023 at 8:22=E2=80=AFAM Kewen.Lin wro= te: > > Hi, > > Similar to the existing function vect_build_gather_load_calls, > this patch is to factor out the handling on scatter store > having gs_info.decl to vect_build_scatter_store_calls which > is a new function. It also does some minor refactoring like > moving some variables' declarations close to their uses and > restrict the scope for some of them etc. > > It's a pre-patch for upcoming vectorizable_store re-structuring > for costing. > > Bootstrapped and regtested on x86_64-redhat-linux, > aarch64-linux-gnu and powerpc64{,le}-linux-gnu. > > Is it ok for trunk? OK. Richard. > Kewen > ----- > > gcc/ChangeLog: > > * tree-vect-stmts.cc (vect_build_scatter_store_calls): New, facto= r > out from ... > (vectorizable_store): ... here. > --- > gcc/tree-vect-stmts.cc | 411 +++++++++++++++++++++-------------------- > 1 file changed, 212 insertions(+), 199 deletions(-) > > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc > index cd8e0a76374..f8a904de503 100644 > --- a/gcc/tree-vect-stmts.cc > +++ b/gcc/tree-vect-stmts.cc > @@ -2989,6 +2989,216 @@ vect_build_gather_load_calls (vec_info *vinfo, st= mt_vec_info stmt_info, > *vec_stmt =3D STMT_VINFO_VEC_STMTS (stmt_info)[0]; > } > > +/* Build a scatter store call while vectorizing STMT_INFO. Insert new > + instructions before GSI and add them to VEC_STMT. GS_INFO describes > + the scatter store operation. If the store is conditional, MASK is th= e > + unvectorized condition, otherwise MASK is null. */ > + > +static void > +vect_build_scatter_store_calls (vec_info *vinfo, stmt_vec_info stmt_info= , > + gimple_stmt_iterator *gsi, gimple **vec_s= tmt, > + gather_scatter_info *gs_info, tree mask) > +{ > + loop_vec_info loop_vinfo =3D dyn_cast (vinfo); > + tree vectype =3D STMT_VINFO_VECTYPE (stmt_info); > + poly_uint64 nunits =3D TYPE_VECTOR_SUBPARTS (vectype); > + int ncopies =3D vect_get_num_copies (loop_vinfo, vectype); > + enum { NARROW, NONE, WIDEN } modifier; > + poly_uint64 scatter_off_nunits > + =3D TYPE_VECTOR_SUBPARTS (gs_info->offset_vectype); > + > + tree perm_mask =3D NULL_TREE, mask_halfvectype =3D NULL_TREE; > + if (known_eq (nunits, scatter_off_nunits)) > + modifier =3D NONE; > + else if (known_eq (nunits * 2, scatter_off_nunits)) > + { > + modifier =3D WIDEN; > + > + /* Currently gathers and scatters are only supported for > + fixed-length vectors. */ > + unsigned int count =3D scatter_off_nunits.to_constant (); > + vec_perm_builder sel (count, count, 1); > + for (unsigned i =3D 0; i < (unsigned int) count; ++i) > + sel.quick_push (i | (count / 2)); > + > + vec_perm_indices indices (sel, 1, count); > + perm_mask =3D vect_gen_perm_mask_checked (gs_info->offset_vectype,= indices); > + gcc_assert (perm_mask !=3D NULL_TREE); > + } > + else if (known_eq (nunits, scatter_off_nunits * 2)) > + { > + modifier =3D NARROW; > + > + /* Currently gathers and scatters are only supported for > + fixed-length vectors. */ > + unsigned int count =3D nunits.to_constant (); > + vec_perm_builder sel (count, count, 1); > + for (unsigned i =3D 0; i < (unsigned int) count; ++i) > + sel.quick_push (i | (count / 2)); > + > + vec_perm_indices indices (sel, 2, count); > + perm_mask =3D vect_gen_perm_mask_checked (vectype, indices); > + gcc_assert (perm_mask !=3D NULL_TREE); > + ncopies *=3D 2; > + > + if (mask) > + mask_halfvectype =3D truth_type_for (gs_info->offset_vectype); > + } > + else > + gcc_unreachable (); > + > + tree rettype =3D TREE_TYPE (TREE_TYPE (gs_info->decl)); > + tree arglist =3D TYPE_ARG_TYPES (TREE_TYPE (gs_info->decl)); > + tree ptrtype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglist= ); > + tree masktype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglis= t); > + tree idxtype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglist= ); > + tree srctype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglist= ); > + tree scaletype =3D TREE_VALUE (arglist); > + > + gcc_checking_assert (TREE_CODE (masktype) =3D=3D INTEGER_TYPE > + && TREE_CODE (rettype) =3D=3D VOID_TYPE); > + > + tree ptr =3D fold_convert (ptrtype, gs_info->base); > + if (!is_gimple_min_invariant (ptr)) > + { > + gimple_seq seq; > + ptr =3D force_gimple_operand (ptr, &seq, true, NULL_TREE); > + class loop *loop =3D LOOP_VINFO_LOOP (loop_vinfo); > + edge pe =3D loop_preheader_edge (loop); > + basic_block new_bb =3D gsi_insert_seq_on_edge_immediate (pe, seq); > + gcc_assert (!new_bb); > + } > + > + tree mask_arg =3D NULL_TREE; > + if (mask =3D=3D NULL_TREE) > + { > + mask_arg =3D build_int_cst (masktype, -1); > + mask_arg =3D vect_init_vector (vinfo, stmt_info, mask_arg, masktyp= e, NULL); > + } > + > + tree scale =3D build_int_cst (scaletype, gs_info->scale); > + > + auto_vec vec_oprnds0; > + auto_vec vec_oprnds1; > + auto_vec vec_masks; > + if (mask) > + { > + tree mask_vectype =3D truth_type_for (vectype); > + vect_get_vec_defs_for_operand (vinfo, stmt_info, > + modifier =3D=3D NARROW ? ncopies / 2= : ncopies, > + mask, &vec_masks, mask_vectype); > + } > + vect_get_vec_defs_for_operand (vinfo, stmt_info, > + modifier =3D=3D WIDEN ? ncopies / 2 : nc= opies, > + gs_info->offset, &vec_oprnds0); > + tree op =3D vect_get_store_rhs (stmt_info); > + vect_get_vec_defs_for_operand (vinfo, stmt_info, > + modifier =3D=3D NARROW ? ncopies / 2 : n= copies, op, > + &vec_oprnds1); > + > + tree vec_oprnd0 =3D NULL_TREE, vec_oprnd1 =3D NULL_TREE; > + tree mask_op =3D NULL_TREE; > + tree src, vec_mask; > + for (int j =3D 0; j < ncopies; ++j) > + { > + if (modifier =3D=3D WIDEN) > + { > + if (j & 1) > + op =3D permute_vec_elements (vinfo, vec_oprnd0, vec_oprnd0, p= erm_mask, > + stmt_info, gsi); > + else > + op =3D vec_oprnd0 =3D vec_oprnds0[j / 2]; > + src =3D vec_oprnd1 =3D vec_oprnds1[j]; > + if (mask) > + mask_op =3D vec_mask =3D vec_masks[j]; > + } > + else if (modifier =3D=3D NARROW) > + { > + if (j & 1) > + src =3D permute_vec_elements (vinfo, vec_oprnd1, vec_oprnd1, > + perm_mask, stmt_info, gsi); > + else > + src =3D vec_oprnd1 =3D vec_oprnds1[j / 2]; > + op =3D vec_oprnd0 =3D vec_oprnds0[j]; > + if (mask) > + mask_op =3D vec_mask =3D vec_masks[j / 2]; > + } > + else > + { > + op =3D vec_oprnd0 =3D vec_oprnds0[j]; > + src =3D vec_oprnd1 =3D vec_oprnds1[j]; > + if (mask) > + mask_op =3D vec_mask =3D vec_masks[j]; > + } > + > + if (!useless_type_conversion_p (srctype, TREE_TYPE (src))) > + { > + gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (src)), > + TYPE_VECTOR_SUBPARTS (srctype))); > + tree var =3D vect_get_new_ssa_name (srctype, vect_simple_var); > + src =3D build1 (VIEW_CONVERT_EXPR, srctype, src); > + gassign *new_stmt =3D gimple_build_assign (var, VIEW_CONVERT_EX= PR, src); > + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); > + src =3D var; > + } > + > + if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) > + { > + gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), > + TYPE_VECTOR_SUBPARTS (idxtype))); > + tree var =3D vect_get_new_ssa_name (idxtype, vect_simple_var); > + op =3D build1 (VIEW_CONVERT_EXPR, idxtype, op); > + gassign *new_stmt =3D gimple_build_assign (var, VIEW_CONVERT_EX= PR, op); > + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); > + op =3D var; > + } > + > + if (mask) > + { > + tree utype; > + mask_arg =3D mask_op; > + if (modifier =3D=3D NARROW) > + { > + tree var > + =3D vect_get_new_ssa_name (mask_halfvectype, vect_simple_= var); > + gassign *new_stmt > + =3D gimple_build_assign (var, > + (j & 1) ? VEC_UNPACK_HI_EXPR > + : VEC_UNPACK_LO_EXPR, > + mask_op); > + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gs= i); > + mask_arg =3D var; > + } > + tree optype =3D TREE_TYPE (mask_arg); > + if (TYPE_MODE (masktype) =3D=3D TYPE_MODE (optype)) > + utype =3D masktype; > + else > + utype =3D lang_hooks.types.type_for_mode (TYPE_MODE (optype),= 1); > + tree var =3D vect_get_new_ssa_name (utype, vect_scalar_var); > + mask_arg =3D build1 (VIEW_CONVERT_EXPR, utype, mask_arg); > + gassign *new_stmt > + =3D gimple_build_assign (var, VIEW_CONVERT_EXPR, mask_arg); > + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); > + mask_arg =3D var; > + if (!useless_type_conversion_p (masktype, utype)) > + { > + gcc_assert (TYPE_PRECISION (utype) <=3D TYPE_PRECISION (mas= ktype)); > + tree var =3D vect_get_new_ssa_name (masktype, vect_scalar_v= ar); > + new_stmt =3D gimple_build_assign (var, NOP_EXPR, mask_arg); > + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gs= i); > + mask_arg =3D var; > + } > + } > + > + gcall *new_stmt > + =3D gimple_build_call (gs_info->decl, 5, ptr, mask_arg, op, src, = scale); > + vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); > + > + STMT_VINFO_VEC_STMTS (stmt_info).safe_push (new_stmt); > + } > + *vec_stmt =3D STMT_VINFO_VEC_STMTS (stmt_info)[0]; > +} > + > /* Prepare the base and offset in GS_INFO for vectorization. > Set *DATAREF_PTR to the loop-invariant base address and *VEC_OFFSET > to the vectorized offset argument for the first copy of STMT_INFO. > @@ -8212,205 +8422,8 @@ vectorizable_store (vec_info *vinfo, > > if (memory_access_type =3D=3D VMAT_GATHER_SCATTER && gs_info.decl) > { > - tree vec_oprnd0 =3D NULL_TREE, vec_oprnd1 =3D NULL_TREE, src; > - tree arglist =3D TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); > - tree rettype, srctype, ptrtype, idxtype, masktype, scaletype; > - tree ptr, var, scale, vec_mask; > - tree mask_arg =3D NULL_TREE, mask_op =3D NULL_TREE, perm_mask =3D = NULL_TREE; > - tree mask_halfvectype =3D mask_vectype; > - edge pe =3D loop_preheader_edge (loop); > - gimple_seq seq; > - basic_block new_bb; > - enum { NARROW, NONE, WIDEN } modifier; > - poly_uint64 scatter_off_nunits > - =3D TYPE_VECTOR_SUBPARTS (gs_info.offset_vectype); > - > - if (known_eq (nunits, scatter_off_nunits)) > - modifier =3D NONE; > - else if (known_eq (nunits * 2, scatter_off_nunits)) > - { > - modifier =3D WIDEN; > - > - /* Currently gathers and scatters are only supported for > - fixed-length vectors. */ > - unsigned int count =3D scatter_off_nunits.to_constant (); > - vec_perm_builder sel (count, count, 1); > - for (i =3D 0; i < (unsigned int) count; ++i) > - sel.quick_push (i | (count / 2)); > - > - vec_perm_indices indices (sel, 1, count); > - perm_mask =3D vect_gen_perm_mask_checked (gs_info.offset_vectyp= e, > - indices); > - gcc_assert (perm_mask !=3D NULL_TREE); > - } > - else if (known_eq (nunits, scatter_off_nunits * 2)) > - { > - modifier =3D NARROW; > - > - /* Currently gathers and scatters are only supported for > - fixed-length vectors. */ > - unsigned int count =3D nunits.to_constant (); > - vec_perm_builder sel (count, count, 1); > - for (i =3D 0; i < (unsigned int) count; ++i) > - sel.quick_push (i | (count / 2)); > - > - vec_perm_indices indices (sel, 2, count); > - perm_mask =3D vect_gen_perm_mask_checked (vectype, indices); > - gcc_assert (perm_mask !=3D NULL_TREE); > - ncopies *=3D 2; > - > - if (mask) > - mask_halfvectype =3D truth_type_for (gs_info.offset_vectype); > - } > - else > - gcc_unreachable (); > - > - rettype =3D TREE_TYPE (TREE_TYPE (gs_info.decl)); > - ptrtype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglist)= ; > - masktype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglist= ); > - idxtype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglist)= ; > - srctype =3D TREE_VALUE (arglist); arglist =3D TREE_CHAIN (arglist)= ; > - scaletype =3D TREE_VALUE (arglist); > - > - gcc_checking_assert (TREE_CODE (masktype) =3D=3D INTEGER_TYPE > - && TREE_CODE (rettype) =3D=3D VOID_TYPE); > - > - ptr =3D fold_convert (ptrtype, gs_info.base); > - if (!is_gimple_min_invariant (ptr)) > - { > - ptr =3D force_gimple_operand (ptr, &seq, true, NULL_TREE); > - new_bb =3D gsi_insert_seq_on_edge_immediate (pe, seq); > - gcc_assert (!new_bb); > - } > - > - if (mask =3D=3D NULL_TREE) > - { > - mask_arg =3D build_int_cst (masktype, -1); > - mask_arg =3D vect_init_vector (vinfo, stmt_info, > - mask_arg, masktype, NULL); > - } > - > - scale =3D build_int_cst (scaletype, gs_info.scale); > - > - auto_vec vec_oprnds0; > - auto_vec vec_oprnds1; > - auto_vec vec_masks; > - if (mask) > - { > - tree mask_vectype =3D truth_type_for (vectype); > - vect_get_vec_defs_for_operand (vinfo, stmt_info, > - modifier =3D=3D NARROW > - ? ncopies / 2 : ncopies, > - mask, &vec_masks, mask_vectype); > - } > - vect_get_vec_defs_for_operand (vinfo, stmt_info, > - modifier =3D=3D WIDEN > - ? ncopies / 2 : ncopies, > - gs_info.offset, &vec_oprnds0); > - vect_get_vec_defs_for_operand (vinfo, stmt_info, > - modifier =3D=3D NARROW > - ? ncopies / 2 : ncopies, > - op, &vec_oprnds1); > - for (j =3D 0; j < ncopies; ++j) > - { > - if (modifier =3D=3D WIDEN) > - { > - if (j & 1) > - op =3D permute_vec_elements (vinfo, vec_oprnd0, vec_oprnd= 0, > - perm_mask, stmt_info, gsi); > - else > - op =3D vec_oprnd0 =3D vec_oprnds0[j / 2]; > - src =3D vec_oprnd1 =3D vec_oprnds1[j]; > - if (mask) > - mask_op =3D vec_mask =3D vec_masks[j]; > - } > - else if (modifier =3D=3D NARROW) > - { > - if (j & 1) > - src =3D permute_vec_elements (vinfo, vec_oprnd1, vec_oprn= d1, > - perm_mask, stmt_info, gsi); > - else > - src =3D vec_oprnd1 =3D vec_oprnds1[j / 2]; > - op =3D vec_oprnd0 =3D vec_oprnds0[j]; > - if (mask) > - mask_op =3D vec_mask =3D vec_masks[j / 2]; > - } > - else > - { > - op =3D vec_oprnd0 =3D vec_oprnds0[j]; > - src =3D vec_oprnd1 =3D vec_oprnds1[j]; > - if (mask) > - mask_op =3D vec_mask =3D vec_masks[j]; > - } > - > - if (!useless_type_conversion_p (srctype, TREE_TYPE (src))) > - { > - gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (src)= ), > - TYPE_VECTOR_SUBPARTS (srctype))); > - var =3D vect_get_new_ssa_name (srctype, vect_simple_var); > - src =3D build1 (VIEW_CONVERT_EXPR, srctype, src); > - gassign *new_stmt > - =3D gimple_build_assign (var, VIEW_CONVERT_EXPR, src); > - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gs= i); > - src =3D var; > - } > - > - if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) > - { > - gcc_assert (known_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op))= , > - TYPE_VECTOR_SUBPARTS (idxtype))); > - var =3D vect_get_new_ssa_name (idxtype, vect_simple_var); > - op =3D build1 (VIEW_CONVERT_EXPR, idxtype, op); > - gassign *new_stmt > - =3D gimple_build_assign (var, VIEW_CONVERT_EXPR, op); > - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gs= i); > - op =3D var; > - } > - > - if (mask) > - { > - tree utype; > - mask_arg =3D mask_op; > - if (modifier =3D=3D NARROW) > - { > - var =3D vect_get_new_ssa_name (mask_halfvectype, > - vect_simple_var); > - gassign *new_stmt > - =3D gimple_build_assign (var, (j & 1) ? VEC_UNPACK_HI= _EXPR > - : VEC_UNPACK_LO_E= XPR, > - mask_op); > - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt= , gsi); > - mask_arg =3D var; > - } > - tree optype =3D TREE_TYPE (mask_arg); > - if (TYPE_MODE (masktype) =3D=3D TYPE_MODE (optype)) > - utype =3D masktype; > - else > - utype =3D lang_hooks.types.type_for_mode (TYPE_MODE (opty= pe), 1); > - var =3D vect_get_new_ssa_name (utype, vect_scalar_var); > - mask_arg =3D build1 (VIEW_CONVERT_EXPR, utype, mask_arg); > - gassign *new_stmt > - =3D gimple_build_assign (var, VIEW_CONVERT_EXPR, mask_arg= ); > - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gs= i); > - mask_arg =3D var; > - if (!useless_type_conversion_p (masktype, utype)) > - { > - gcc_assert (TYPE_PRECISION (utype) > - <=3D TYPE_PRECISION (masktype)); > - var =3D vect_get_new_ssa_name (masktype, vect_scalar_va= r); > - new_stmt =3D gimple_build_assign (var, NOP_EXPR, mask_a= rg); > - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt= , gsi); > - mask_arg =3D var; > - } > - } > - > - gcall *new_stmt > - =3D gimple_build_call (gs_info.decl, 5, ptr, mask_arg, op, sr= c, scale); > - vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); > - > - STMT_VINFO_VEC_STMTS (stmt_info).safe_push (new_stmt); > - } > - *vec_stmt =3D STMT_VINFO_VEC_STMTS (stmt_info)[0]; > + vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, > + &gs_info, mask); > return true; > } > else if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) >=3D 3) > -- > 2.31.1