From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by sourceware.org (Postfix) with ESMTPS id AA5B03858C2C for ; Tue, 4 Jul 2023 03:14:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org AA5B03858C2C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688440461; x=1719976461; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=/79XyqAkSDQf+G0jEZkKsPTYwCGISDJo7rdHC86Kt0U=; b=kBdcIi7lyKX2v5QHc86gRBdyiZ1sxu1h9/Fq455fcwsxDb1cmyl0DbQh P7bG72cNrx7YSuij5SfvLkmiJv1N9W1DNd6qJixRnNNA8j6a8OnADuigf 1owjyUopWDMbdyMrBqzOIuS2FqFmUJm30gwWgpmDb+jxKci2Q49twRvyr IGTmB7McWbfft6ymkkcPHydM4CPnX+XHyhFg+HPjlPR4NsBsfw440nlXp RWXDB0z/oSt+96XKOuKkb9iCsRkazjYNCJ8v6snnSzbc4jnN08jbuGx/E oBtkvCdXX4kZ1D3ddT+NfWYoKI8LkTjZh/1yJRqDgULUieMD2vVydtSpu g==; X-IronPort-AV: E=McAfee;i="6600,9927,10760"; a="352846380" X-IronPort-AV: E=Sophos;i="6.01,179,1684825200"; d="scan'208";a="352846380" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 20:14:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10760"; a="892719680" X-IronPort-AV: E=Sophos;i="6.01,179,1684825200"; d="scan'208";a="892719680" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga005.jf.intel.com with ESMTP; 03 Jul 2023 20:14:20 -0700 Received: from fmsmsx611.amr.corp.intel.com (10.18.126.91) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 3 Jul 2023 20:14:19 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx611.amr.corp.intel.com (10.18.126.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 3 Jul 2023 20:14:19 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27 via Frontend Transport; Mon, 3 Jul 2023 20:14:19 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.47) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Mon, 3 Jul 2023 20:14:19 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X5YhlvhVbqxz/VmmiSDjb+W3aov2VCJ7yo0Uvr354y9V0OKLvCAemomhXva7hRVbslge1V7V7Qmvvo2unPNWYM90es69q/nZgG7llKWVG1hWUgsMa1/f2Dp6FtlbLdwllRzh9J4/yDFZWpdP8WWEroLP/nVDulJu+MV2FaDDqmaoquSrIMnZTLfKNWHllwPxSKHEbG0suycIt4xORcWcokTnh+Q9i3Y/SchiTq4FUngUNGBfGxf0IenT2OXnK8OizGi1SEIsyfYwSVpXjp5yox7HvlYp+tHPGrkCs4zp9oZA57tISNYGUKkZ5rq0zdaks4suWrZouAube5OIFg0GLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DZ/sub3fGMhuGnTl7p2SQA/EgfF9KTb1HHX1Hym/49I=; b=TnGlALT7Sr/jXYONIFuAFcMR4JawRqsgDk696f0XWsWstwhxVoYMqie1Wre+oGl/WYmcl577/WZ2QKTptM+KDnDrFOpsh/7n05wYtSBnKxHwG2qx+r1rT7Oynyx9+IAEvp31mRSRfcYfxAuoR84ErJcCuanMzzVgdiNvs9KEIoBKxZ3ljVJKBxW576ncR+yUJwfY88w428f3HezJuBZgN/fyjt9BBgsVYlceNAllDaSPKNbzw5xnk8EseQRmDQf5wXLP/x5QCWwkCnbDHWdefUrK/AjGZSmLSxDBWjCHopdaPi5WDVC+lFbaZYv9BFzwDDqH/MOFV5IIlxCHMoCnzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from MW5PR11MB5908.namprd11.prod.outlook.com (2603:10b6:303:194::10) by PH8PR11MB7023.namprd11.prod.outlook.com (2603:10b6:510:221::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 03:14:16 +0000 Received: from MW5PR11MB5908.namprd11.prod.outlook.com ([fe80::127c:f4cc:e699:8f73]) by MW5PR11MB5908.namprd11.prod.outlook.com ([fe80::127c:f4cc:e699:8f73%6]) with mapi id 15.20.6544.024; Tue, 4 Jul 2023 03:14:16 +0000 From: "Li, Pan2" To: Richard Sandiford , "juzhe.zhong@rivai.ai" CC: "gcc-patches@gcc.gnu.org" , "rguenther@suse.de" Subject: RE: [PATCH V7] Machine Description: Add LEN_MASK_{GATHER_LOAD, SCATTER_STORE} pattern Thread-Topic: [PATCH V7] Machine Description: Add LEN_MASK_{GATHER_LOAD, SCATTER_STORE} pattern Thread-Index: AQHZrZ7lL24NzBF6H0mKE7lbgeB6qa+oD2OTgADgRWA= Date: Tue, 4 Jul 2023 03:14:16 +0000 Message-ID: References: <20230703110912.15311-1-juzhe.zhong@rivai.ai> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: MW5PR11MB5908:EE_|PH8PR11MB7023:EE_ x-ms-office365-filtering-correlation-id: c844e718-2150-45a7-4c85-08db7c3cc005 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: sozJ9WxOv0/BKKgtwA3N0C9BKcZXxGd+UxLl4KViwyabsoBtg78K1uxWztx1yMBVLhVB1eV72rsWx5wMlAR7I6LZ5eQYcWffKALTiQk5j+y27OTvPjGVLLRRqVdhomm+deTcxdM1suG2uGfmlflIrnGiazGhICpUGLbODTXGP/u/HMFJST9m+mnuDKpKlTZiSrGT4fO6lkd/oRPCi6BEEl0G5JQ/gp9kX8bHeuRXru98fz9RKXeeFHGxWms1HSwMJB+Tlin5UNVqBMNn3P8PeaYZAR49jmoUJ2lT+B+ZuhZUEgsqTtVArYRzhFnAHKjqQ4ilHnHLZESevtVxe+306Qzkgc1M7D5MghYX6sc3HUqGPj75Lb/peM/+axpIVvtHPgjQudiHV+gyaRMLvm1OedqstdtbE6cAbL6Y/Fz7fXql+whcpXNK6X4DeRodqRXuHkHpxjpQaOAYzZCckcyR43av8Tn38CVpELOEH1kxeMKIZ0h23Ub/LRSWYXRyRC19xmWcyHLVfhDsTpxX9wg4xmaSKfOmqtFc0ISGB3iDzcwA8uPfjSuDHQ99h9oPTCcItTE5+4YHpAyRErsO7SRG5XlR28O4yGEEpR/vgFeGzXI= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW5PR11MB5908.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(136003)(376002)(396003)(366004)(346002)(39860400002)(451199021)(66446008)(66556008)(66946007)(64756008)(66476007)(4326008)(76116006)(478600001)(316002)(2906002)(8676002)(33656002)(8936002)(52536014)(5660300002)(41300700001)(54906003)(86362001)(55016003)(966005)(9686003)(110136005)(38070700005)(30864003)(7696005)(38100700002)(53546011)(186003)(6506007)(26005)(82960400001)(71200400001)(122000001)(83380400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?lebq96716wi4C/oHBeAThzvk9B5UxcO2UbC2W95wqkcgnfvpufcqdpHLv4jy?= =?us-ascii?Q?D+MLYGu2wdks+jyre4yGesgauhZpHBG0+myt83w4JsbIzHzC9FTeyaDZAfRf?= =?us-ascii?Q?qScWzrGQLns15fMHs0WJTj2yhR9vDcKwUbMR75p7oCMtbMgG58pxmLwoiFVQ?= =?us-ascii?Q?LReEUMXwjCPcACK+FblgFcx4tefQRG8ersGc3AXCpz3NynT92jx4tpx5okeo?= =?us-ascii?Q?D7iWglFVM/S5NaPQROcm6MKq9joMY3WtNoGwpGKBkK7FJ6goq3v/lphESJBf?= =?us-ascii?Q?Q92id57DPfZ776Sw6tm8CZuRFpiviVeVoZyw/OP5TJQ8XO0XkHzFbwNRzyuH?= =?us-ascii?Q?K9BY828miCq9PIHmteneKWJd7eA4rNqqNjI96d+uNEHd/kboJie8uzgsiyJn?= =?us-ascii?Q?iTM66FDhH0XrS8MauAX1Q/co5wAnBnrj/oqnYBrEwcNQF8167cCIsrz0lpca?= =?us-ascii?Q?HWaDWriFmehPhiYO5fg1nt7JjmfgiIs3PW33TZr7zzPO96m2KMTLHsVjB9NI?= =?us-ascii?Q?l7hUtReE5s4/mg/h24GUXe/a7fOQ03JLv11HDgCG4U4Fdp4YRVe6E+k1+Q0U?= =?us-ascii?Q?xUAculuD/tPOSPkXrVdZJmLuSk9zBzwemmfiLAXweJyUAoA6VDdU9+cVlvAp?= =?us-ascii?Q?3Qpn2Hr03Gzv/8lPT4ejhlijTx4k1viibgy7KEXljP5UUr3gNoeKA4LXfGue?= =?us-ascii?Q?/5q2F1d2GJKmllBYa1F05OXzq4h7JzJBrNPfgwUbwBT/SE3PGCirJx4q1NHs?= =?us-ascii?Q?2rnHOw58aOLZauMMfzGbf4TrTI/eT8QHvEZulQwKB+92/OCO0xKcLYBbGgFH?= =?us-ascii?Q?LvvmhnUZ47xf6/TNqP2vl/PvHRSsqx1xqORbyD/LfcWDx3W3TYFWCUMZjBxB?= =?us-ascii?Q?cW6jNta9u38FFpWPN+w/uu3b2LU0I7RhsnIRgIlfdwhG+Ci8ATwMXMJrEa5Z?= =?us-ascii?Q?tf6UbkwIxgQGerjoJsTsMc8/h5VPJvYmvHYI//7xjeYSrZj8wob7yigd/UBI?= =?us-ascii?Q?D7NmtX3MeciS/xtfAOlMmN8pjF/KwXhBCRlpTR3rCelgwd35aLcQAP9Kgk1d?= =?us-ascii?Q?ugkY8hu5uCiqr7n7xIr6db8Mfzamj0jtoyj5uBXMR9yA/2wDcmX8JduqzmeT?= =?us-ascii?Q?EZNlEs/UQ9zrbQZljc8AuJSX1mLm2ffncibHQ+qT8ovVdQcb08KTC5WqaBxo?= =?us-ascii?Q?sCA1kCL+I8LFIVfub8IBg7pxuMCNLKuWs5/a2+p7rcNZCDeSpXVeBNIKqeoU?= =?us-ascii?Q?y779X7LtVeSENBZ2EcyHK99U38sG3ow+Hjcmv0DLUGQRsY68wIPLYlfDJbWS?= =?us-ascii?Q?SSFlVHpxAwUyZXldW0HhZplsWcrlhpmNiibeethMsveY6KkFLMw/3lSVHwOd?= =?us-ascii?Q?gFqG7ISa4nZZeDQp1LKPzmdudq4j4/4HlhYbJewonlClWWlxDlZs/LrQJ8rH?= =?us-ascii?Q?6nK4/wTWsX6R8jiGozqyrALjtfq02O8/5auV3g2u+MtPP8uwl5yiXpxi8fjD?= =?us-ascii?Q?wVq5mtP4Lq3UzLf5uOcUQNlgIwQgKfw7YR0n7zkzz0RYGlXBMJz+eGUr0MOq?= =?us-ascii?Q?/SAoZ67plXiD4zMoAik=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5908.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: c844e718-2150-45a7-4c85-08db7c3cc005 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jul 2023 03:14:16.4154 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: eKe2uDKjirZ8KpM/LjXkudKlgsTkHc+m+kOlObrL2tGZ+8Ivv7smEXGZmLmDvbrum0o6fQVfoxG10Mey2j4tDA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7023 X-OriginatorOrg: intel.com X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,KAM_SHORT,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Committed as both the bootstrap and regression tests passed, thanks Richard= . Pan -----Original Message----- From: Gcc-patches On = Behalf Of Richard Sandiford via Gcc-patches Sent: Monday, July 3, 2023 9:50 PM To: juzhe.zhong@rivai.ai Cc: gcc-patches@gcc.gnu.org; rguenther@suse.de Subject: Re: [PATCH V7] Machine Description: Add LEN_MASK_{GATHER_LOAD, SCA= TTER_STORE} pattern juzhe.zhong@rivai.ai writes: > From: Ju-Zhe Zhong > > Hi, Richi and Richard. > > Base one the review comments from Richard: > https://gcc.gnu.org/pipermail/gcc-patches/2023-July/623405.html > > I change len_mask_gather_load/len_mask_scatter_store order into: > {len,bias,mask} > > We adjust adding len and mask using using add_len_and_mask_args > which is same as partial_load/parial_store. > > Now, the codes become more reasonable and easier maintain. > > This patch is adding LEN_MASK_{GATHER_LOAD,SCATTER_STORE} to allow target= s > handle flow control by mask and loop control by length on gather/scatter = memory > operations. Consider this following case: > > #include > void > f (uint8_t *restrict a, > uint8_t *restrict b, int n, > int base, int step, > int *restrict cond) > { > for (int i =3D 0; i < n; ++i) > { > if (cond[i]) > a[i * step + base] =3D b[i * step + base]; > } > } > > We hope RVV can vectorize such case into following IR: > > loop_len =3D SELECT_VL > control_mask =3D comparison > v =3D LEN_MASK_GATHER_LOAD (.., loop_len, bias, control_mask) > LEN_SCATTER_STORE (... v, ..., loop_len, bias, control_mask) > > This patch doesn't apply such patterns into vectorizer, just add patterns > and update the documents. > > Will send patch which apply such patterns into vectorizer soon after this > patch is approved. > > Ok for trunk? > > gcc/ChangeLog: > > * doc/md.texi: Add len_mask_gather_load/len_mask_scatter_store. > * internal-fn.cc (expand_scatter_store_optab_fn): Ditto. > (expand_gather_load_optab_fn): Ditto. > (internal_load_fn_p): Ditto. > (internal_store_fn_p): Ditto. > (internal_gather_scatter_fn_p): Ditto. > (internal_fn_len_index): Ditto. > (internal_fn_mask_index): Ditto. > (internal_fn_stored_value_index): Ditto. > * internal-fn.def (LEN_MASK_GATHER_LOAD): Ditto. > (LEN_MASK_SCATTER_STORE): Ditto. > * optabs.def (OPTAB_CD): Ditto. Nice! OK, thanks. Richard > --- > gcc/doc/md.texi | 17 +++++++++++++++++ > gcc/internal-fn.cc | 32 +++++++++++++++++--------------- > gcc/internal-fn.def | 8 ++++++-- > gcc/optabs.def | 2 ++ > 4 files changed, 42 insertions(+), 17 deletions(-) > > diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi > index 5e5482265cd..f14dd32b2dc 100644 > --- a/gcc/doc/md.texi > +++ b/gcc/doc/md.texi > @@ -5040,6 +5040,15 @@ operand 5. Bit @var{i} of the mask is set if elem= ent @var{i} > of the result should be loaded from memory and clear if element @var{i} > of the result should be set to zero. > =20 > +@cindex @code{len_mask_gather_load@var{m}@var{n}} instruction pattern > +@item @samp{len_mask_gather_load@var{m}@var{n}} > +Like @samp{gather_load@var{m}@var{n}}, but takes an extra length operand= (operand 5), > +a bias operand (operand 6) as well as a mask operand (operand 7). Simil= ar to len_maskload, > +the instruction loads at most (operand 5 + operand 6) elements from memo= ry. > +Bit @var{i} of the mask is set if element @var{i} of the result should > +be loaded from memory and clear if element @var{i} of the result should = be undefined. > +Mask elements @var{i} with @var{i} > (operand 5 + operand 6) are ignored= . > + > @cindex @code{scatter_store@var{m}@var{n}} instruction pattern > @item @samp{scatter_store@var{m}@var{n}} > Store a vector of mode @var{m} into several distinct memory locations. > @@ -5069,6 +5078,14 @@ Like @samp{scatter_store@var{m}@var{n}}, but takes= an extra mask operand as > operand 5. Bit @var{i} of the mask is set if element @var{i} > of the result should be stored to memory. > =20 > +@cindex @code{len_mask_scatter_store@var{m}@var{n}} instruction pattern > +@item @samp{len_mask_scatter_store@var{m}@var{n}} > +Like @samp{scatter_store@var{m}@var{n}}, but takes an extra length opera= nd (operand 5), > +a bias operand (operand 6) as well as a mask operand (operand 7). The i= nstruction stores > +at most (operand 5 + operand 6) elements of (operand 4) to memory. > +Bit @var{i} of the mask is set if element @var{i} of (operand 4) should = be stored. > +Mask elements @var{i} with @var{i} > (operand 5 + operand 6) are ignored= . > + > @cindex @code{vec_set@var{m}} instruction pattern > @item @samp{vec_set@var{m}} > Set given field in the vector value. Operand 0 is the vector to modify, > diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc > index c1fcb38b17b..303df102d81 100644 > --- a/gcc/internal-fn.cc > +++ b/gcc/internal-fn.cc > @@ -3507,7 +3507,6 @@ expand_scatter_store_optab_fn (internal_fn, gcall *= stmt, direct_optab optab) > { > internal_fn ifn =3D gimple_call_internal_fn (stmt); > int rhs_index =3D internal_fn_stored_value_index (ifn); > - int mask_index =3D internal_fn_mask_index (ifn); > tree base =3D gimple_call_arg (stmt, 0); > tree offset =3D gimple_call_arg (stmt, 1); > tree scale =3D gimple_call_arg (stmt, 2); > @@ -3518,19 +3517,14 @@ expand_scatter_store_optab_fn (internal_fn, gcall= *stmt, direct_optab optab) > HOST_WIDE_INT scale_int =3D tree_to_shwi (scale); > rtx rhs_rtx =3D expand_normal (rhs); > =20 > - class expand_operand ops[6]; > + class expand_operand ops[8]; > int i =3D 0; > create_address_operand (&ops[i++], base_rtx); > create_input_operand (&ops[i++], offset_rtx, TYPE_MODE (TREE_TYPE (off= set))); > create_integer_operand (&ops[i++], TYPE_UNSIGNED (TREE_TYPE (offset)))= ; > create_integer_operand (&ops[i++], scale_int); > create_input_operand (&ops[i++], rhs_rtx, TYPE_MODE (TREE_TYPE (rhs)))= ; > - if (mask_index >=3D 0) > - { > - tree mask =3D gimple_call_arg (stmt, mask_index); > - rtx mask_rtx =3D expand_normal (mask); > - create_input_operand (&ops[i++], mask_rtx, TYPE_MODE (TREE_TYPE (m= ask))); > - } > + i =3D add_len_and_mask_args (ops, i, stmt); > =20 > insn_code icode =3D convert_optab_handler (optab, TYPE_MODE (TREE_TYPE= (rhs)), > TYPE_MODE (TREE_TYPE (offset))); > @@ -3553,18 +3547,13 @@ expand_gather_load_optab_fn (internal_fn, gcall *= stmt, direct_optab optab) > HOST_WIDE_INT scale_int =3D tree_to_shwi (scale); > =20 > int i =3D 0; > - class expand_operand ops[6]; > + class expand_operand ops[8]; > create_output_operand (&ops[i++], lhs_rtx, TYPE_MODE (TREE_TYPE (lhs))= ); > create_address_operand (&ops[i++], base_rtx); > create_input_operand (&ops[i++], offset_rtx, TYPE_MODE (TREE_TYPE (off= set))); > create_integer_operand (&ops[i++], TYPE_UNSIGNED (TREE_TYPE (offset)))= ; > create_integer_operand (&ops[i++], scale_int); > - if (optab =3D=3D mask_gather_load_optab) > - { > - tree mask =3D gimple_call_arg (stmt, 4); > - rtx mask_rtx =3D expand_normal (mask); > - create_input_operand (&ops[i++], mask_rtx, TYPE_MODE (TREE_TYPE (m= ask))); > - } > + i =3D add_len_and_mask_args (ops, i, stmt); > insn_code icode =3D convert_optab_handler (optab, TYPE_MODE (TREE_TYPE= (lhs)), > TYPE_MODE (TREE_TYPE (offset))); > expand_insn (icode, i, ops); > @@ -4415,6 +4404,7 @@ internal_load_fn_p (internal_fn fn) > case IFN_MASK_LOAD_LANES: > case IFN_GATHER_LOAD: > case IFN_MASK_GATHER_LOAD: > + case IFN_LEN_MASK_GATHER_LOAD: > case IFN_LEN_LOAD: > case IFN_LEN_MASK_LOAD: > return true; > @@ -4436,6 +4426,7 @@ internal_store_fn_p (internal_fn fn) > case IFN_MASK_STORE_LANES: > case IFN_SCATTER_STORE: > case IFN_MASK_SCATTER_STORE: > + case IFN_LEN_MASK_SCATTER_STORE: > case IFN_LEN_STORE: > case IFN_LEN_MASK_STORE: > return true; > @@ -4454,8 +4445,10 @@ internal_gather_scatter_fn_p (internal_fn fn) > { > case IFN_GATHER_LOAD: > case IFN_MASK_GATHER_LOAD: > + case IFN_LEN_MASK_GATHER_LOAD: > case IFN_SCATTER_STORE: > case IFN_MASK_SCATTER_STORE: > + case IFN_LEN_MASK_SCATTER_STORE: > return true; > =20 > default: > @@ -4477,6 +4470,10 @@ internal_fn_len_index (internal_fn fn) > case IFN_LEN_MASK_STORE: > return 2; > =20 > + case IFN_LEN_MASK_GATHER_LOAD: > + case IFN_LEN_MASK_SCATTER_STORE: > + return 4; > + > default: > return -1; > } > @@ -4502,6 +4499,10 @@ internal_fn_mask_index (internal_fn fn) > case IFN_LEN_MASK_STORE: > return 4; > =20 > + case IFN_LEN_MASK_GATHER_LOAD: > + case IFN_LEN_MASK_SCATTER_STORE: > + return 6; > + > default: > return (conditional_internal_fn_code (fn) !=3D ERROR_MARK > || get_unconditional_internal_fn (fn) !=3D IFN_LAST ? 0 : -1); > @@ -4520,6 +4521,7 @@ internal_fn_stored_value_index (internal_fn fn) > case IFN_MASK_STORE_LANES: > case IFN_SCATTER_STORE: > case IFN_MASK_SCATTER_STORE: > + case IFN_LEN_MASK_SCATTER_STORE: > return 3; > =20 > case IFN_LEN_STORE: > diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def > index d9fcca8430f..9b73e540d55 100644 > --- a/gcc/internal-fn.def > +++ b/gcc/internal-fn.def > @@ -48,14 +48,14 @@ along with GCC; see the file COPYING3. If not see > - mask_load: currently just maskload > - load_lanes: currently just vec_load_lanes > - mask_load_lanes: currently just vec_mask_load_lanes > - - gather_load: used for {mask_,}gather_load > + - gather_load: used for {mask_,len_mask,}gather_load > - len_load: currently just len_load > - len_maskload: currently just len_maskload > =20 > - mask_store: currently just maskstore > - store_lanes: currently just vec_store_lanes > - mask_store_lanes: currently just vec_mask_store_lanes > - - scatter_store: used for {mask_,}scatter_store > + - scatter_store: used for {mask_,len_mask,}scatter_store > - len_store: currently just len_store > - len_maskstore: currently just len_maskstore > =20 > @@ -157,6 +157,8 @@ DEF_INTERNAL_OPTAB_FN (MASK_LOAD_LANES, ECF_PURE, > DEF_INTERNAL_OPTAB_FN (GATHER_LOAD, ECF_PURE, gather_load, gather_load) > DEF_INTERNAL_OPTAB_FN (MASK_GATHER_LOAD, ECF_PURE, > mask_gather_load, gather_load) > +DEF_INTERNAL_OPTAB_FN (LEN_MASK_GATHER_LOAD, ECF_PURE, > + len_mask_gather_load, gather_load) > =20 > DEF_INTERNAL_OPTAB_FN (LEN_LOAD, ECF_PURE, len_load, len_load) > DEF_INTERNAL_OPTAB_FN (LEN_MASK_LOAD, ECF_PURE, len_maskload, len_masklo= ad) > @@ -164,6 +166,8 @@ DEF_INTERNAL_OPTAB_FN (LEN_MASK_LOAD, ECF_PURE, len_m= askload, len_maskload) > DEF_INTERNAL_OPTAB_FN (SCATTER_STORE, 0, scatter_store, scatter_store) > DEF_INTERNAL_OPTAB_FN (MASK_SCATTER_STORE, 0, > mask_scatter_store, scatter_store) > +DEF_INTERNAL_OPTAB_FN (LEN_MASK_SCATTER_STORE, 0, > + len_mask_scatter_store, scatter_store) > =20 > DEF_INTERNAL_OPTAB_FN (MASK_STORE, 0, maskstore, mask_store) > DEF_INTERNAL_OPTAB_FN (STORE_LANES, ECF_CONST, vec_store_lanes, store_la= nes) > diff --git a/gcc/optabs.def b/gcc/optabs.def > index a901b68c538..73c9a0c760f 100644 > --- a/gcc/optabs.def > +++ b/gcc/optabs.def > @@ -95,8 +95,10 @@ OPTAB_CD(len_maskload_optab, "len_maskload$a$b") > OPTAB_CD(len_maskstore_optab, "len_maskstore$a$b") > OPTAB_CD(gather_load_optab, "gather_load$a$b") > OPTAB_CD(mask_gather_load_optab, "mask_gather_load$a$b") > +OPTAB_CD(len_mask_gather_load_optab, "len_mask_gather_load$a$b") > OPTAB_CD(scatter_store_optab, "scatter_store$a$b") > OPTAB_CD(mask_scatter_store_optab, "mask_scatter_store$a$b") > +OPTAB_CD(len_mask_scatter_store_optab, "len_mask_scatter_store$a$b") > OPTAB_CD(vec_extract_optab, "vec_extract$a$b") > OPTAB_CD(vec_init_optab, "vec_init$a$b")