From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by sourceware.org (Postfix) with ESMTPS id D3E453858D28 for ; Fri, 7 Apr 2023 03:23:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D3E453858D28 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680837828; x=1712373828; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=2zM+sCnYprcP5PeAIg+MAdKoYKstY2w5ylP/G6jQBU0=; b=aLyDUUDuTd2Mpju+zx39q/bFzDJDr5rRcrtPlueStDiGPHgbVUOzKoFA xqfqSzBGbLKHBoL9jnirInmdj8WiUK0tiaKtHSqiIkQeTxDkEPc7GCBdU x6hdkgL3flbaMx5ifKSFEEPvCOz/KCKcz50xtS197Qh8SCOn4Zwf0aKK1 O5cclp4aqpRnYTeLgIlt9NoT5kt20NvJ3sDLFmVUeyaum5QY+KX0yPKlh NA7jATFs5hO/AbzZ5tRvV+DHfDmXv+dmYMK1qoHE51a1BlR/9bGULj5k3 D05lB90Ive3XlKYaOLcHC5NbnuNZWTcCK5N290TddSiPd6plWT1pE3xki A==; X-IronPort-AV: E=McAfee;i="6600,9927,10672"; a="344689358" X-IronPort-AV: E=Sophos;i="5.98,324,1673942400"; d="scan'208";a="344689358" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Apr 2023 20:23:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10672"; a="831050898" X-IronPort-AV: E=Sophos;i="5.98,324,1673942400"; d="scan'208";a="831050898" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by fmsmga001.fm.intel.com with ESMTP; 06 Apr 2023 20:23:46 -0700 Received: from orsmsx612.amr.corp.intel.com (10.22.229.25) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Thu, 6 Apr 2023 20:23:46 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx612.amr.corp.intel.com (10.22.229.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21 via Frontend Transport; Thu, 6 Apr 2023 20:23:46 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.104) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.21; Thu, 6 Apr 2023 20:23:46 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Kx9QW3V3xInAlqOpG5JQFxUBCmuo8veGpNXjBPYz8U2rm7KTlK4ujnTplvliRG0CeYAvurXZ19LiB3G6QoH2u/XbrQ2iCvTz/F8MWVGD/xkgENHmO9Fz0MgTT4eSoQ0oCTGeSv8l5RDcfU3p48cDVEqifZDhnG58z9189vbQA1U/9UD55qSsuBmpub7DInqCMsK5Oz+tC3gixtzFXEcB3BrYyHCxzlJSWX5er2Ulje4bb6XkMikFGH8/kFwFPqUKKNO4spQXDyyESSeP09WKmRZYWtpmjLr59iadxvwmz8+cwInvab4jIZhBelfUpVARN22i1gOfrZjLA5dtRdnvVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=te7/JxzTj8Z6pK9eE3n/0uQzVLs2AFrciycXqmm91Jw=; b=DfbPWYzvkUSWCwZT21MkxfLQxV6lvYeZKnVDuIMx/l9KkgQ9gmhcwY9PxFD8gp0OC/GJMPlxDgmLpoddTv7isGzl6b/4aYBldk6cl3ayphPoFQTFa6AtolZawyTGbbi9vrxhOXMDxtJkDj+jOhpNGQaTZdeq23et6xEeskrE5E1tJZP/uRXjC99HaU+R6bXZ88ochNTVQ30ZcgOuse/kABsIpZv3Tp1bHEC40YhcTECNiy7Eyn7sjfB/JEkMv4qGInMPnwj6VgiWQzsiuPbX0ZdB73tAq9iNX+kc6240GVAYFTdM1+YlqVNJ1PQznmQ/NSu6aa4zImruj2e0al1KBQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from MW5PR11MB5908.namprd11.prod.outlook.com (2603:10b6:303:194::10) by SJ0PR11MB7702.namprd11.prod.outlook.com (2603:10b6:a03:4e2::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6277.31; Fri, 7 Apr 2023 03:23:44 +0000 Received: from MW5PR11MB5908.namprd11.prod.outlook.com ([fe80::d84:4cba:8307:78e8]) by MW5PR11MB5908.namprd11.prod.outlook.com ([fe80::d84:4cba:8307:78e8%4]) with mapi id 15.20.6277.031; Fri, 7 Apr 2023 03:23:44 +0000 From: "Li, Pan2" To: "juzhe.zhong@rivai.ai" , "gcc-patches@gcc.gnu.org" CC: "richard.sandiford@arm.com" , "rguenther@suse.de" , "jeffreyalaw@gmail.com" Subject: RE: [PATCH] VECT: Add WHILE_LEN pattern for decrement IV support for auto-vectorization Thread-Topic: [PATCH] VECT: Add WHILE_LEN pattern for decrement IV support for auto-vectorization Thread-Index: AQHZaPMRCydpPTOGnEi3nE7J45ppY68fLf5A Date: Fri, 7 Apr 2023 03:23:44 +0000 Message-ID: References: <20230407014741.139387-1-juzhe.zhong@rivai.ai> In-Reply-To: <20230407014741.139387-1-juzhe.zhong@rivai.ai> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: MW5PR11MB5908:EE_|SJ0PR11MB7702:EE_ x-ms-office365-filtering-correlation-id: 2e0ca602-a06d-44df-33ae-08db37177e21 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: hnnfO2ppCR9bzbCbeqtutu2vxu2dqs4YYjcYWlmqqhNb83CIcQ/w9TwlUITOih3PdxSE9xsh/Ojk7kpLWmEOdjx3TgIoPKUH1RAHhLMQpBQzHJlmfdDWQdzaMam185z4gO8v9JgjL2sLvZaatpS4nWuXalRHHOL0RLktK7IwPvCX2LIj1po54sRepFjCX5uPneKitfi60b6ruNoZQMFSVSSS5X3sEE3dNT6w0mK9DUMEyuu6QjcvWyyuWois7rcOc9FrUa+pTSCX2+JDmt/TYbcLkdjeayjIo4dBx55i56DxDXEIWdc577p7H0PjfFN7SeCT3DpI3kzkvm2Mi6Nr8QhOSScjYrae0TyFC5++iFNgoVWCynHbpksTS1xAli5vfXCuRvaqQgkfsBnh2THnHZXwyUwypNRk6aG1Yg95Oew60I1fzywEposNSjEs4MFnvUQDENrJ7tOobla237BSHMEM9y9E/ObZGaQCI/8iIKcCAXnLzxBNiFSbR2cep8W1uFNVCFvvSpIWAbKevsqOJTzIDs2R4DqgdGcYWApQE/XeTbQ1E4t/Xu1gSNn+wViQXXWsux2Zq0Kxdv5NXLIZ29L5bkSygA5OibYP2lYDZ+Y= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW5PR11MB5908.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(396003)(376002)(346002)(366004)(39860400002)(136003)(451199021)(54906003)(110136005)(316002)(86362001)(83380400001)(82960400001)(66446008)(66946007)(64756008)(66556008)(76116006)(33656002)(478600001)(66476007)(38100700002)(38070700005)(122000001)(4326008)(8676002)(186003)(53546011)(2906002)(6506007)(41300700001)(26005)(9686003)(71200400001)(30864003)(55016003)(8936002)(7696005)(52536014)(5660300002)(966005)(579004);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?17TllpfiTz3DvQMQFtgBmuz5XvXOxv85CfoQ5uT7UL/mwsWTDNeaQP9N9Jbr?= =?us-ascii?Q?tNRGzChT2uOUuFVcL9skWABw9gADsNb/J9yJD2AQxiEYWkSrGWKso3kBKqyB?= =?us-ascii?Q?kHpbfJnXMyJA+BpWKitKk6qyTnFAEXJZiqKw/KGa+uz0DhtP22wXZ+rPdo1K?= =?us-ascii?Q?DBBDglqkqxKahyUigfSpX9ze6Rf4jkE9wgiNUDFB/cgQwAVbNJZ3SP/S1TS6?= =?us-ascii?Q?igx5e0gekkQOrqs0WsQnWVqbI6FN/HeKmlotZQx2wMdoemNzfEkdfB6OAer6?= =?us-ascii?Q?2jwihsOpYlWynNCxdMSy+OIFmMBsRMuGRH58Se12MH1bA1D8Xbatxy5F5E1F?= =?us-ascii?Q?iFAhnABcaDdMx1kRskgJoorYBO1eOVMcEcIoTxIyBpDWmfRypULsb9JZ7kNK?= =?us-ascii?Q?JoqL3F/03GI8IGkD/ZxvCkK/ZQ9vtMSldal/mlYKYwbzebslNfDLOTOtCxQV?= =?us-ascii?Q?tOfAOKnuOOMYirWGDeG+KwMUEwZTsv0qBpOCyDhKjP7HLBnzr8D/b5tfrg2A?= =?us-ascii?Q?j5ucJm3oLOBxeHDCPJSC/ZthLKT5olGFoxWbR/qqKAigp5TDoc+nF8C/w6g1?= =?us-ascii?Q?koho4+F3NKanOjq2lGTwsYhAW+NuzUN+pvmffmWCFfUgtpzTxVOwo/JXmfN0?= =?us-ascii?Q?N2GAdih8xpau748B2IjXVnG8IaZmjFh5lTVurvXVNNK7luzTKANLLSDuRwq9?= =?us-ascii?Q?s3t/IhNrub3pv9OcF7wWP9c5fIte/XHCeBHrFn9NZfr487z8iUXfHywKSO77?= =?us-ascii?Q?8ANI2G/0ESN0YFR2B/1y7dczDfef5Ps5xRDToQQlMseTvxT8OEdaQodXL5X3?= =?us-ascii?Q?zjyuD+4gL2UkcyBPd7t8SVp1OJOeAy10ol5bEeF+tqmsmFqPUW4WRtb/x953?= =?us-ascii?Q?HvlGj4QBgXeldDxBw9eNE//dM2LJXOCFckiFTL5/oS5TnPw3x/46QDPBYpsQ?= =?us-ascii?Q?YQ/u26+8InatYDeUaNSXr9o7FYo8NTzw1oO9Pz6IVclk//WX7r+eWPhKGujv?= =?us-ascii?Q?OBPjawpoGSua+X+NUa37WJ/L9Tm0cs3QdkEfW4ruw8e4tgng8Pg2oL3hggCP?= =?us-ascii?Q?YMA7zvIWgAlFsNIszAe3sR+U+wBfBW4szZWvZqS7dOdKQGuRclBQNAGolG9e?= =?us-ascii?Q?zzdC73JCKerFH+eYVXMhzwAHXTS3sHz0KoGeNG0JumUthogorIB37bPl+ulp?= =?us-ascii?Q?SxpGLb/HZR64YV4mz5mOtZH/cyAc/5XN4O3MiGLd9H5Fv40OT3SKL183meuw?= =?us-ascii?Q?2hAq4LS9PcUPzlbg2FFBQd2y87eWsvYToQDGDDh038iqnojJTI/8o3PO8Qs/?= =?us-ascii?Q?Qr+bG+CLAm+T6Q1Q9NNsUKW3+2VEi5x7F+G3QN1FF3/P6/4aRrfn9nV54Ws1?= =?us-ascii?Q?BSdUcT9jtYBz4D6tMR/6d+cBmqZCMp9yMMvkkyug3Kqd/MY5c66R5mtrPti3?= =?us-ascii?Q?p5PRtFTIzqduHkfGkLVbh1JvWUJDp0hrIUKsxS5scY1RVTHA4BPp8evo4cMQ?= =?us-ascii?Q?e2TqWVpSvvtNW3i/qV/Eg9Q5Jzz2EUgylGKXOxYTGkC5rpU0EchSAH3wfVJn?= =?us-ascii?Q?TqdnT+ZRF/J9wWnk9X4=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MW5PR11MB5908.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2e0ca602-a06d-44df-33ae-08db37177e21 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Apr 2023 03:23:44.2908 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: eTym8ak+LdFJBbBJ5nGSdxjXv/KHBE39QBm7X8PQWbqeHdygCwnAzKa3OE3huM+nT0uUaBegu4JLVOj0nWkaKA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB7702 X-OriginatorOrg: intel.com X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: The bootstrap in X86 passed with this patch applied, target commit id a8c83= 51cf4fedb842988eed4f73304019c361e86 (13.0.1 20230407). Pan -----Original Message----- From: Gcc-patches On = Behalf Of juzhe.zhong@rivai.ai Sent: Friday, April 7, 2023 9:48 AM To: gcc-patches@gcc.gnu.org Cc: richard.sandiford@arm.com; rguenther@suse.de; jeffreyalaw@gmail.com; Ju= zhe-Zhong Subject: [PATCH] VECT: Add WHILE_LEN pattern for decrement IV support for a= uto-vectorization From: Juzhe-Zhong This patch is to add WHILE_LEN pattern. It's inspired by RVV ISA simple "vvaddint32.s" example: https://github.com/riscv/riscv-v-spec/blob/master/example/vvaddint32.s More details are in "vect_set_loop_controls_by_while_len" implementation an= d comments. Consider such following case: #define N 16 int src[N]; int dest[N]; void foo (int n) { for (int i =3D 0; i < n; i++) dest[i] =3D src[i]; } -march=3Drv64gcv -O3 --param riscv-autovec-preference=3Dscalable -fno-vect-= cost-model -fno-tree-loop-distribute-patterns: foo: =20 ble a0,zero,.L1 lui a4,%hi(.LANCHOR0) addi a4,a4,%lo(.LANCHOR0) addi a3,a4,64 csrr a2,vlenb .L3: vsetvli a5,a0,e32,m1,ta,ma vle32.v v1,0(a4) sub a0,a0,a5 vse32.v v1,0(a3) add a4,a4,a2 add a3,a3,a2 bne a0,zero,.L3 .L1: ret gcc/ChangeLog: * doc/md.texi: Add WHILE_LEN support. * internal-fn.cc (while_len_direct): Ditto. (expand_while_len_optab_fn): Ditto. (direct_while_len_optab_supported_p): Ditto. * internal-fn.def (WHILE_LEN): Ditto. * optabs.def (OPTAB_D): Ditto. * tree-ssa-loop-manip.cc (create_iv): Ditto. * tree-ssa-loop-manip.h (create_iv): Ditto. * tree-vect-loop-manip.cc (vect_set_loop_controls_by_while_len): Di= tto. (vect_set_loop_condition_partial_vectors): Ditto. * tree-vect-loop.cc (vect_get_loop_len): Ditto. * tree-vect-stmts.cc (vectorizable_store): Ditto. (vectorizable_load): Ditto. * tree-vectorizer.h (vect_get_loop_len): Ditto. --- gcc/doc/md.texi | 14 +++ gcc/internal-fn.cc | 29 ++++++ gcc/internal-fn.def | 1 + gcc/optabs.def | 1 + gcc/tree-ssa-loop-manip.cc | 4 +- gcc/tree-ssa-loop-manip.h | 2 +- gcc/tree-vect-loop-manip.cc | 186 ++++++++++++++++++++++++++++++++++-- gcc/tree-vect-loop.cc | 35 +++++-- gcc/tree-vect-stmts.cc | 9 +- gcc/tree-vectorizer.h | 4 +- 10 files changed, 264 insertions(+), 21 deletions(-) diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index 8e3113599fd..72178ab01= 4c 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -4965,6 +4965,20 @@ for (i =3D 1; i < operand3; i++) operand0[i] =3D operand0[i - 1] && (operand1 + i < operand2); @end smal= lexample =20 +@cindex @code{while_len@var{m}@var{n}} instruction pattern @item=20 +@code{while_len@var{m}@var{n}} Set operand 0 to the number of active=20 +elements in vector will be updated value. +operand 1 is the total elements need to be updated value. +operand 2 is the vectorization factor. +The operation is equivalent to: + +@smallexample +operand0 =3D MIN (operand1, operand2); +operand2 can be const_poly_int or poly_int related to vector mode size. +Some target like RISC-V has a standalone instruction to get MIN (n,=20 +MODE SIZE) so that we can reduce a use of general purpose register. +@end smallexample + @cindex @code{check_raw_ptrs@var{m}} instruction pattern @item @samp{chec= k_raw_ptrs@var{m}} Check whether, given two pointers @var{a} and @var{b} a= nd a length @var{len}, diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc= index 6e81dc05e0e..5f44def90d3 100644 --- a/gcc/internal-fn.cc +++ b/gcc/internal-fn.cc @@ -127,6 +127,7 @@ init_internal_fns () #define cond_binary_direct { 1, 1= , true } #define cond_ternary_direct { 1, 1, true } #define while_direct = { 0, 2, false } +#define while_len_direct { 0, 0, false } #define fold_extract_direct { 2, 2, false } #define fold_left_direct { 1,= 1, false } #define mask_fold_left_direct { 1, 1, false } @@ -3702,6 +3703= ,33 @@ expand_while_optab_fn (internal_fn, gcall *stmt, convert_optab optab= ) emit_move_insn (lhs_rtx, ops[0].value); } =20 +/* Expand WHILE_LEN call STMT using optab OPTAB. */ static void=20 +expand_while_len_optab_fn (internal_fn, gcall *stmt, convert_optab=20 +optab) { + expand_operand ops[3]; + tree rhs_type[2]; + + tree lhs =3D gimple_call_lhs (stmt); + tree lhs_type =3D TREE_TYPE (lhs); + rtx lhs_rtx =3D expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE); =20 + create_output_operand (&ops[0], lhs_rtx, TYPE_MODE (lhs_type)); + + for (unsigned int i =3D 0; i < gimple_call_num_args (stmt); ++i) + { + tree rhs =3D gimple_call_arg (stmt, i); + rhs_type[i] =3D TREE_TYPE (rhs); + rtx rhs_rtx =3D expand_normal (rhs); + create_input_operand (&ops[i + 1], rhs_rtx, TYPE_MODE (rhs_type[i]))= ; + } + + insn_code icode =3D direct_optab_handler (optab, TYPE_MODE=20 + (rhs_type[0])); + + expand_insn (icode, 3, ops); + if (!rtx_equal_p (lhs_rtx, ops[0].value)) + emit_move_insn (lhs_rtx, ops[0].value); } + /* Expand a call to a convert-like optab using the operands in STMT. FN has a single output operand and NARGS input operands. */ =20 @@ -3843,6 +3871,7 @@ multi_vector_optab_supported_p (convert_optab optab, = tree_pair types, #define direct_scatter_store_optab_supported_p convert_op= tab_supported_p #define direct_len_store_optab_supported_p direct_optab_su= pported_p #define direct_while_optab_supported_p convert_optab_supported_p +#define direct_while_len_optab_supported_p direct_optab_supported_p #define direct_fold_extract_optab_supported_p direct_optab_supported_p #d= efine direct_fold_left_optab_supported_p direct_optab_supported_p #define = direct_mask_fold_left_optab_supported_p direct_optab_supported_p diff --git= a/gcc/internal-fn.def b/gcc/internal-fn.def index 7fe742c2ae7..3a933abff5d= 100644 --- a/gcc/internal-fn.def +++ b/gcc/internal-fn.def @@ -153,6 +153,7 @@ DEF_INTERNAL_OPTAB_FN (VEC_SET, 0, vec_set, vec_set) D= EF_INTERNAL_OPTAB_FN (LEN_STORE, 0, len_store, len_store) =20 DEF_INTERNAL_OPTAB_FN (WHILE_ULT, ECF_CONST | ECF_NOTHROW, while_ult, whil= e) +DEF_INTERNAL_OPTAB_FN (WHILE_LEN, ECF_CONST | ECF_NOTHROW, while_len,=20 +while_len) DEF_INTERNAL_OPTAB_FN (CHECK_RAW_PTRS, ECF_CONST | ECF_NOTHROW, check_raw_ptrs, check_ptrs) DEF_INTERNAL_OPTAB_FN (CHECK_WAR_PTRS, ECF_CONST | ECF_NOTHROW, diff --git= a/gcc/optabs.def b/gcc/optabs.def index 695f5911b30..f5938bd2c24 100644 --- a/gcc/optabs.def +++ b/gcc/optabs.def @@ -476,3 +476,4 @@ OPTAB_DC (vec_series_optab, "vec_series$a", VEC_SERIES)= OPTAB_D (vec_shl_insert_optab, "vec_shl_insert_$a") OPTAB_D (len_load_op= tab, "len_load_$a") OPTAB_D (len_store_optab, "len_store_$a") +OPTAB_D (while_len_optab, "while_len$a") diff --git a/gcc/tree-ssa-loop-manip.cc b/gcc/tree-ssa-loop-manip.cc index = 09acc1c94cc..cdbf280e249 100644 --- a/gcc/tree-ssa-loop-manip.cc +++ b/gcc/tree-ssa-loop-manip.cc @@ -59,14 +59,14 @@ static bitmap_obstack loop_renamer_obstack; void crea= te_iv (tree base, tree step, tree var, class loop *loop, gimple_stmt_iterator *incr_pos, bool after, - tree *var_before, tree *var_after) + tree *var_before, tree *var_after, enum tree_code code) { gassign *stmt; gphi *phi; tree initial, step1; gimple_seq stmts; tree vb, va; - enum tree_code incr_op =3D PLUS_EXPR; + enum tree_code incr_op =3D code; edge pe =3D loop_preheader_edge (loop); =20 if (var !=3D NULL_TREE) diff --git a/gcc/tree-ssa-loop-manip.h b/gcc/tree-ssa-loop-manip.h index d4= 9273a3987..da755320a3a 100644 --- a/gcc/tree-ssa-loop-manip.h +++ b/gcc/tree-ssa-loop-manip.h @@ -23,7 +23,7 @@ along with GCC; see the file COPYING3. If not see typed= ef void (*transform_callback)(class loop *, void *); =20 extern void create_iv (tree, tree, tree, class loop *, gimple_stmt_iterato= r *, - bool, tree *, tree *); + bool, tree *, tree *, enum tree_code =3D PLUS_EXPR); extern void rewrite_into_loop_closed_ssa (bitmap, unsigned); extern void = verify_loop_closed_ssa (bool, class loop * =3D NULL); =20 diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc inde= x f60fa50e8f4..f3cd6c51d2e 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -682,6 +682,173 @@ vect_set_loop_controls_directly (class loop *loop, lo= op_vec_info loop_vinfo, return next_ctrl; } =20 +/* Helper for vect_set_loop_condition_partial_vectors. Generate definitio= ns + for all the rgroup controls in RGC and return a control that is nonzero + when the loop needs to iterate. Add any new preheader statements to + PREHEADER_SEQ. Use LOOP_COND_GSI to insert code before the exit gcond. + + RGC belongs to loop LOOP. The loop originally iterated NITERS + times and has been vectorized according to LOOP_VINFO. + + Unlike vect_set_loop_controls_directly which is iterating from 0-based = IV + to TEST_LIMIT - bias. + + In vect_set_loop_controls_by_while_len, we are iterating from start at + IV =3D TEST_LIMIT - bias and keep subtract IV by the length calculated = by + IFN_WHILE_LEN pattern. + + Note: the cost of the code generated by this function is modeled + by vect_estimate_min_profitable_iters, so changes here may need + corresponding changes there. + + 1. Single rgroup, the Gimple IR should be: + + + _19 =3D (unsigned long) n_5(D); + ... + + : + ... + # ivtmp_20 =3D PHI + ... + _22 =3D .WHILE_LEN (ivtmp_20, vf); + ... + vector statement (use _22); + ... + ivtmp_21 =3D ivtmp_20 - _22; + ... + if (ivtmp_21 !=3D 0) + goto ; [75.00%] + else + goto ; [25.00%] + + + return; + + Note: IFN_WHILE_LEN will guarantee "ivtmp_21 =3D ivtmp_20 - _22" never + underflow 0. + + 2. Multiple rgroup, the Gimple IR should be: + + + _70 =3D (unsigned long) bnd.7_52; + _71 =3D _70 * 2; + _72 =3D MAX_EXPR <_71, 4>; + _73 =3D _72 + 18446744073709551612; + ... + + : + ... + # ivtmp_74 =3D PHI + # ivtmp_77 =3D PHI + _76 =3D .WHILE_LEN (ivtmp_74, vf * nitems_per_ctrl); + _79 =3D .WHILE_LEN (ivtmp_77, vf * nitems_per_ctrl); + ... + vector statement (use _79); + ... + vector statement (use _76); + ... + _65 =3D _79 / 2; + vector statement (use _65); + ... + _68 =3D _76 / 2; + vector statement (use _68); + ... + ivtmp_78 =3D ivtmp_77 - _79; + ivtmp_75 =3D ivtmp_74 - _76; + ... + if (ivtmp_78 !=3D 0) + goto ; [75.00%] + else + goto ; [25.00%] + + + return; + +*/ + +static tree +vect_set_loop_controls_by_while_len (class loop *loop, loop_vec_info loop_= vinfo, + gimple_seq *preheader_seq, + gimple_seq *header_seq, + rgroup_controls *rgc, tree niters) { + tree compare_type =3D LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo); + tree iv_type =3D LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo); + /* We are not allowing masked approach in WHILE_LEN. */ + gcc_assert (!LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)); + + tree ctrl_type =3D rgc->type; + unsigned int nitems_per_iter =3D rgc->max_nscalars_per_iter *=20 + rgc->factor; + poly_uint64 nitems_per_ctrl =3D TYPE_VECTOR_SUBPARTS (ctrl_type) *=20 + rgc->factor; + poly_uint64 vf =3D LOOP_VINFO_VECT_FACTOR (loop_vinfo); + + /* Calculate the maximum number of item values that the rgroup + handles in total, the number that it handles for each iteration + of the vector loop. */ + tree nitems_total =3D niters; + if (nitems_per_iter !=3D 1) + { + /* We checked before setting LOOP_VINFO_USING_PARTIAL_VECTORS_P that + these multiplications don't overflow. */ + tree compare_factor =3D build_int_cst (compare_type, nitems_per_iter= ); + nitems_total =3D gimple_build (preheader_seq, MULT_EXPR, compare_typ= e, + nitems_total, compare_factor); + } + + /* Convert the comparison value to the IV type (either a no-op or + a promotion). */ + nitems_total =3D gimple_convert (preheader_seq, iv_type, nitems_total); + + /* Create an induction variable that counts the number of items + processed. */ + tree index_before_incr, index_after_incr; gimple_stmt_iterator=20 + incr_gsi; bool insert_after; standard_iv_increment_position (loop,=20 + &incr_gsi, &insert_after); + + /* Test the decremented IV, which will never underflow 0 since we have + IFN_WHILE_LEN to gurantee that. */ tree test_limit =3D=20 + nitems_total; + + /* Provide a definition of each control in the group. */ + tree ctrl; + unsigned int i; + FOR_EACH_VEC_ELT_REVERSE (rgc->controls, i, ctrl) + { + /* Previous controls will cover BIAS items. This control covers the + next batch. */ + poly_uint64 bias =3D nitems_per_ctrl * i; + tree bias_tree =3D build_int_cst (iv_type, bias); + + /* Rather than have a new IV that starts at TEST_LIMIT and goes down= to + BIAS, prefer to use the same TEST_LIMIT - BIAS based IV for each + control and adjust the bound down by BIAS. */ + tree this_test_limit =3D test_limit; + if (i !=3D 0) + { + this_test_limit =3D gimple_build (preheader_seq, MAX_EXPR, iv_type, + this_test_limit, bias_tree); + this_test_limit =3D gimple_build (preheader_seq, MINUS_EXPR, iv_type, + this_test_limit, bias_tree); + } + + /* Create decrement IV. */ + create_iv (this_test_limit, ctrl, NULL_TREE, loop, &incr_gsi, + insert_after, &index_before_incr, &index_after_incr, + MINUS_EXPR); + + poly_uint64 final_vf =3D vf * nitems_per_iter; + tree vf_step =3D build_int_cst (iv_type, final_vf); + tree res_len =3D gimple_build (header_seq, IFN_WHILE_LEN, iv_type, + index_before_incr, vf_step); + gassign *assign =3D gimple_build_assign (ctrl, res_len); + gimple_seq_add_stmt (header_seq, assign); + } + + return index_after_incr; +} + /* Set up the iteration condition and rgroup controls for LOOP, given that LOOP_VINFO_USING_PARTIAL_VECTORS_P is true for the vectorized loop. LOOP_VINFO describes the vectorization of LOOP. NITERS is @@ -7= 03,6 +870,7 @@ vect_set_loop_condition_partial_vectors (class loop *loop, =20 bool use_masks_p =3D LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); tree compare_type =3D LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo); + tree iv_type =3D LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo); unsigned int compare_precision =3D TYPE_PRECISION (compare_type); tree orig_niters =3D niters; =20 @@ -757,12 +925,18 @@ vect_set_loop_condition_partial_vectors (class loop *= loop, bool might_wrap_p =3D vect_rgroup_iv_might_wrap_p (loop_vinfo, rgc); =20 /* Set up all controls for this group. */ - test_ctrl =3D vect_set_loop_controls_directly (loop, loop_vinfo, - &preheader_seq, - &header_seq, - loop_cond_gsi, rgc, - niters, niters_skip, - might_wrap_p); + if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type, + OPTIMIZE_FOR_SPEED)) + test_ctrl + =3D vect_set_loop_controls_by_while_len (loop, loop_vinfo, + &preheader_seq, &header_seq, + rgc, niters); + else + test_ctrl + =3D vect_set_loop_controls_directly (loop, loop_vinfo, &preheader_seq= , + &header_seq, loop_cond_gsi, rgc, + niters, niters_skip, + might_wrap_p); } =20 /* Emit all accumulated statements. */ diff --git a/gcc/tree-vect-loop.= cc b/gcc/tree-vect-loop.cc index 1ba9f18d73e..5bffd9a6322 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -10360,12 +10360,14 @@ vect_record_loop_len (loop_vec_info loop_vinfo, v= ec_loop_lens *lens, rgroup that operates on NVECTORS vectors, where 0 <=3D INDEX < NVECTORS= . */ =20 tree -vect_get_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens, - unsigned int nvectors, unsigned int index) +vect_get_loop_len (gimple_stmt_iterator *gsi, loop_vec_info loop_vinfo, + vec_loop_lens *lens, unsigned int nvectors, tree vectype, + unsigned int index) { rgroup_controls *rgl =3D &(*lens)[nvectors - 1]; - bool use_bias_adjusted_len =3D - LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) !=3D 0; + bool use_bias_adjusted_len + =3D LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) !=3D 0; tree=20 + iv_type =3D LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo); =20 /* Populate the rgroup's len array, if this is the first time we've used it. */ @@ -10386,8 +10388,8 @@ vect_get_loop_len (loop_vec_info loop_vinfo, vec_lo= op_lens *lens, if (use_bias_adjusted_len) { gcc_assert (i =3D=3D 0); - tree adjusted_len =3D - make_temp_ssa_name (len_type, NULL, "adjusted_loop_len"); + tree adjusted_len + =3D make_temp_ssa_name (len_type, NULL, "adjusted_loop_len"); SSA_NAME_DEF_STMT (adjusted_len) =3D gimple_build_nop (); rgl->bias_adjusted_ctrl =3D adjusted_len; } @@ -10396,6 +10398,27 @@ vect_get_loop_len (loop_vec_info loop_vinfo, vec_l= oop_lens *lens, =20 if (use_bias_adjusted_len) return rgl->bias_adjusted_ctrl; + else if (direct_internal_fn_supported_p (IFN_WHILE_LEN, iv_type, + OPTIMIZE_FOR_SPEED)) + { + tree loop_len =3D rgl->controls[index]; + poly_int64 nunits1 =3D TYPE_VECTOR_SUBPARTS (rgl->type); + poly_int64 nunits2 =3D TYPE_VECTOR_SUBPARTS (vectype); + if (maybe_ne (nunits1, nunits2)) + { + /* A loop len for data type X can be reused for data type Y + if X has N times more elements than Y and if Y's elements + are N times bigger than X's. */ + gcc_assert (multiple_p (nunits1, nunits2)); + unsigned int factor =3D exact_div (nunits1, nunits2).to_constant (); + gimple_seq seq =3D NULL; + loop_len =3D gimple_build (&seq, RDIV_EXPR, iv_type, loop_len, + build_int_cst (iv_type, factor)); + if (seq) + gsi_insert_seq_before (gsi, seq, GSI_SAME_STMT); + } + return loop_len; + } else return rgl->controls[index]; } diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index efa2d0da= a52..708c8a1d806 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -8653,8 +8653,9 @@ vectorizable_store (vec_info *vinfo, else if (loop_lens) { tree final_len - =3D vect_get_loop_len (loop_vinfo, loop_lens, - vec_num * ncopies, vec_num * j + i); + =3D vect_get_loop_len (gsi, loop_vinfo, loop_lens, + vec_num * ncopies, vectype, + vec_num * j + i); tree ptr =3D build_int_cst (ref_type, align * BITS_PER_UNIT); machine_mode vmode =3D TYPE_MODE (vectype); opt_machine_mode new_ovmode @@ -10009,8 +10010,8 @@ vectorizable_load (vec_info *vinfo, else if (loop_lens && memory_access_type !=3D VMAT_INVARIANT) { tree final_len - =3D vect_get_loop_len (loop_vinfo, loop_lens, - vec_num * ncopies, + =3D vect_get_loop_len (gsi, loop_vinfo, loop_lens, + vec_num * ncopies, vectype, vec_num * j + i); tree ptr =3D build_int_cst (ref_type, align * BITS_PER_UNIT); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 9cf2fb23fe= 3..e5cf38caf4b 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -2293,8 +2293,8 @@ extern tree vect_get_loop_mask (gimple_stmt_iterator = *, vec_loop_masks *, unsigned int, tree, unsigned int); extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned= int, tree, unsigned int); -extern tree vect_get_loop_len (loop_vec_info, vec_loop_lens *, unsigned in= t, - unsigned int); +extern tree vect_get_loop_len (gimple_stmt_iterator *, loop_vec_info, + vec_loop_lens *, unsigned int, tree, unsigned int); extern gimple_seq vect_gen_len (tree, tree, tree, tree); extern stmt_vec_= info info_for_reduction (vec_info *, stmt_vec_info); extern bool reduction= _fn_for_scalar_code (code_helper, internal_fn *); -- 2.36.3