From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on2049.outbound.protection.outlook.com [40.107.6.49]) by sourceware.org (Postfix) with ESMTPS id B0F1338708C4 for ; Mon, 6 Nov 2023 07:39:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B0F1338708C4 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org B0F1338708C4 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=40.107.6.49 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256402; cv=pass; b=EXNH9zfxC3X324d7tx5EXvK8RgVZForqyeJqns72y3zbl+c5oOkSkZRGBi9dte0BYb33Jq8N/4i1vBXqDfU6i9k984KuIWV5KkbH6pfHXxFBv+tylp54WhMoB05HLKCoxCkgL6yhxgSeqh5x3Att4rt/BRXB7pWJiK9ksXJ5ecI= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256402; c=relaxed/simple; bh=82ilZK/QliO2pMxARS+nv+6lHwwtHDkA/4z4ZOroHu4=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=KQ3AfMSqvyHqYvBuJher1if3P3DR4vrE6Gkbx5Hk5X8QGveyqFYAF+8qQr3rDQXeT0dnaQiFvyn8GzOfXV2aDMqsB7ImGQe8UzPSO/C7E5MOg4u94x894XtiIy+HvS9Un5L00e4LKRCvrJTJPug6oSbCwIDoaX0vNEU/VNQyXjk= ARC-Authentication-Results: i=3; server2.sourceware.org ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=edpq2HRGcGctNzPWqegoVBUzvGUqMd9StXmPoI2yD8V7BllhuXEjewigxSf5Zf8WhiBasOS9CHH4N4oIM8jhcHJ4x65g41jpo2V2gFGt20avFHTcm2RzJbNBtIArej1rl99BxgdDpu2YLcKgPCuNAh8aAdKei3n0kfxRn0tcBv1Qf3o/ukf8LGTFtAvvhzXg06GpbW97Oz+FzNPzuzD+u31fV3M5chFqzC9+wEJLD8W5GqieKv4BgP7r8OyyZqrH3mPFLHNCA0/sdOwz0yCMBNTBWvp1dhfzoujAe4YKysVFqHixtJH8R/Gy2awFdBWoJwNSEGRIihX2/KbXFjwp7Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/TIUNAJ+PyfL3OP2qIi1DiyW/YTAPLXWA+lBC3levjE=; b=ZfxgyVHwObG8Dv41aXFrnMaSt03hs122y6UA2iJw53wiA3fPxenzRFDCpdnhF0v5spDOzZ9t77+zS6JSxa3VfpecC0GYr7NW9Blbd1RpGTpw6M6/VwhBTv5dZo5/usudwI81efgQf7n1+icQRsqOgiVSdTVl6aXPwU8sOSwwQ35QDDQe/cCcUJa5DmXSvxeBju8FrQxRMOOK4MJVAwNabHK9Cv13noArWIsZrkj8OTCyLzAfSUCMkccXiifPR2+pBP64GzDW8AFPxiF5iZhozll2q2D0Ose2EYIvfKPtaTq+RNGUOobss+xH3WXMqzx4q7bDB0GwumBf0RRb+TfhPw== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/TIUNAJ+PyfL3OP2qIi1DiyW/YTAPLXWA+lBC3levjE=; b=P0YK4RdZvfPjYALfUTGYb0ytdxBwfb/GRpnsPNFSBJVCCVVqNF57F22bSzJG5xXmKlILVELpsdt0EY+0gqO5EM+syK0ncHyjlsfI1UJviJS1X8dh93O3pYGxYh813eKLQGPPXfb3znLD+X0Gg06M7LnCiB2sF5QPceA09xskxcw= Received: from AM7PR03CA0021.eurprd03.prod.outlook.com (2603:10a6:20b:130::31) by DB9PR08MB9946.eurprd08.prod.outlook.com (2603:10a6:10:3d2::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.27; Mon, 6 Nov 2023 07:39:56 +0000 Received: from AMS1EPF00000040.eurprd04.prod.outlook.com (2603:10a6:20b:130:cafe::50) by AM7PR03CA0021.outlook.office365.com (2603:10a6:20b:130::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28 via Frontend Transport; Mon, 6 Nov 2023 07:39:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AMS1EPF00000040.mail.protection.outlook.com (10.167.16.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.16 via Frontend Transport; Mon, 6 Nov 2023 07:39:55 +0000 Received: ("Tessian outbound 20615a7e7970:v228"); Mon, 06 Nov 2023 07:39:55 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 3b28dcd139ebe858 X-CR-MTA-TID: 64aa7808 Received: from 79fb54de656e.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id A33263C1-A55F-48B4-B5F8-02F0656A5D3E.1; Mon, 06 Nov 2023 07:39:48 +0000 Received: from EUR04-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 79fb54de656e.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 06 Nov 2023 07:39:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lNHBOMk1vI1sHQGBXhtKnYWTvNx8Mo3IepK9L1Ug46d/pjdDop3Nso5NrswNYzk6vgALETt57hAawzxbYHmrRIEq1nKMw2mMvkbrrBXt+bNk+8Jbm/wY/1o31HcnCKSgzrFzL7DsOqfqu8YLBL2iSLHiz/mmIBmG+WCzCYHmbH4W4bF4ctVro5NFluLX2yG1cXgLZRYZ9lUFnTj7A4GOrEOwGr7wLpwM7qKPxxAZ3UGY8KxuHr1w1ibmcskSQh7WT1kQLZaf9uVbPuh705XNAxPv5IdE7u0GMZS4XtaqPw2E/vRa+9QtQm2qsTaxxWqAw3fxR+vU8RZkq3hGrvVQhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/TIUNAJ+PyfL3OP2qIi1DiyW/YTAPLXWA+lBC3levjE=; b=mlNoq5E0vAwHx37KMiIEvf4RxhAKg9PByJ+951iG6AsgFVmuHmjs2NgDFY9eGkW4PEkAF/wONrfbs9fBCnnL63VIuujHSzEpqpaPrGdP/hUUPEzGGr2dpF1QWVT44cd+SNFPfYLPz1zYKrw/RtokWABs8PrFuNmBSY8lTm10f1v84YMkkye6P2T4MZiZU05pVoIS+tVQjW5P1mWJBmY8Xp2z3fFrrePZevUGWiYhxmem8EvXezI2gio+rVZ4XyLbChPMWt3x22hPsN8GL5kdMIpW5gw9YmvaMsa3hxON6IL3gwzdSlKidNGWaz4hktN73WzZ2Vv4Fbqard6qvaR1hg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/TIUNAJ+PyfL3OP2qIi1DiyW/YTAPLXWA+lBC3levjE=; b=P0YK4RdZvfPjYALfUTGYb0ytdxBwfb/GRpnsPNFSBJVCCVVqNF57F22bSzJG5xXmKlILVELpsdt0EY+0gqO5EM+syK0ncHyjlsfI1UJviJS1X8dh93O3pYGxYh813eKLQGPPXfb3znLD+X0Gg06M7LnCiB2sF5QPceA09xskxcw= Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) by DBBPR08MB10676.eurprd08.prod.outlook.com (2603:10a6:10:530::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28; Mon, 6 Nov 2023 07:39:46 +0000 Received: from VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0]) by VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0%5]) with mapi id 15.20.6954.028; Mon, 6 Nov 2023 07:39:45 +0000 Date: Mon, 6 Nov 2023 07:39:42 +0000 From: Tamar Christina To: gcc-patches@gcc.gnu.org Cc: nd@arm.com, rguenther@suse.de, jlaw@ventanamicro.com Subject: [PATCH 9/21]middle-end: implement vectorizable_early_exit for codegen of exit code Message-ID: Content-Type: multipart/mixed; boundary="lnLy9OEdlB9dFHYy" Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO4P123CA0544.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:319::9) To VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) MIME-Version: 1.0 X-MS-TrafficTypeDiagnostic: VI1PR08MB5325:EE_|DBBPR08MB10676:EE_|AMS1EPF00000040:EE_|DB9PR08MB9946:EE_ X-MS-Office365-Filtering-Correlation-Id: d55596e5-ae35-4273-c7e7-08dbde9b922f x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: ewZ6J8Fa860rIegIU5sWO7Oecl1lrvddjAjMgdNMolFpDnEAoCXVcX+7b10UGZ0rS7sKSvhZgI0nah+P+f1fTXQsXwJtXhDNvHqL+E7puHEA+uXjO9uL/QWhOMtyKKyVCD8r5S5eRMrZ1iExYmCwkMXTaH/znBvTm5tAOcgpryHVi5HFOmnHlBlQHjiJ7uLqXJbgpGbUaQPsj0syDHYAiCFIaCF2Gy44HDoY34WJUZBBnv/yOsWM0KKh4AJ/19nkpc+1nsGY9pFxux4T89Nbvtcn2dK1BaJ0Xf+BaoktUaG+SpnAqIDdCWF5Y3aZC4qHgpE7dXEXbsYwinIFeyd1Nr1T22nMmqN4R8bzBrn9CM0bQmzO4/Gi1zeMUpUE6+Inu+L7WpjMAHp8pcIUKvsv4zURgGxJqxYegzLULF5mjHiHsCBHTS8T7kTl0cjN+fyEpMOFyYG4BIuKX7WII5pdQczjGtz2Ufcjs2DrmBsdXyPMeBKg+0J2uXtyfbIVoiyR2Hgqk4H9NQVCJ2dbm16JWEQvuhYRxOaQYDyQ8AhsDS78bNg4H0gAhOdlk8qLdqV8/HKu7KhxNmrsRtDuEj7JmYGfx9IOui/h9xei9kFY83oTDpSEngsEQhORBBUCFnCM X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR08MB5325.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(396003)(376002)(366004)(346002)(39860400002)(230922051799003)(186009)(64100799003)(1800799009)(451199024)(6916009)(4326008)(8676002)(8936002)(41300700001)(36756003)(2906002)(235185007)(38100700002)(5660300002)(6486002)(66556008)(6512007)(316002)(66476007)(86362001)(44832011)(4743002)(66946007)(83380400001)(26005)(2616005)(6666004)(6506007)(478600001)(33964004)(44144004)(2700100001);DIR:OUT;SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB10676 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AMS1EPF00000040.eurprd04.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: bd273281-88fa-4148-cf2d-08dbde9b8c45 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HLqzUB+GD5Dw+G1lPVflynVztfw5cBPcvU1uFJOITtBnqVci2kGeznmoezoYYf5b591yaCXJbawwndPIRtelcBEU/jBCM6D0/v8a3QqDAr1lbjRDOWpcRD7wCnVJm8/2G5LRAdyUYBZlUEgN8AVs0A3SdUYxKrl9U2IMAng++UottBw4X/AgwiV08rRfJPQvJJIXz/aSLDcYz9/dxYcxjASPSP54j7aF/+5D15llyOhIHSLTse/6th/3BQ5DQW0D9wqs1MdvXsGStlG2aeW1SBZP4AT8rSkbwSJTyDjV3KhLw8AEuca0tVxTYHKyX3ovvaBF8i0mD5lCYRI6TNvr+0KkqTawwM2S7Vx6bJCD6KHbiWzVgXKKOjoV9Gm4esyxAmpCoP8G+q9+pDrnzD1KIdjFg74ncMhNhBWpPV+PcDgl0xWn63QcWWdOHsjbFjU9yCpbDXl6d3tus5Cmr/0im6x/40J/2aQhpgHgWyhGmGz5HQ2ULj1Z9sUQ+08imhCZ+6of/UYuw2zCryuTcNKVUiz3Cmji/DOfXjDd90IsjFDBD2MXhjHdBGfX1vVb8GLXbYF634i9aWOjUSiG6uWMc2htu7uq98dqTfhAdYbVqf+BD1DuI/0vt6PJQn3lPPPM6o8woOAsr7Lf5iCS1C8+ldwKiW5xl3gHfDU2oNqzK6HIp+1eF8/+MzT6cw1EGSpO2dTVG4fuXSOfFFWdAh+PdBHbOf5xXrPqtqdoJvjDuX7cRWlHyHSMfwJPdfFqKt0xx+Yrh1XnzsuHS8gPPbG37g== X-Forefront-Antispam-Report: CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(39860400002)(346002)(376002)(136003)(230922051799003)(1800799009)(451199024)(82310400011)(64100799003)(186009)(40470700004)(36840700001)(46966006)(40480700001)(4326008)(86362001)(235185007)(81166007)(356005)(83380400001)(8936002)(47076005)(2906002)(36860700001)(41300700001)(8676002)(5660300002)(4743002)(40460700003)(336012)(82740400003)(70206006)(70586007)(107886003)(6512007)(44832011)(2616005)(33964004)(316002)(6916009)(44144004)(6506007)(6666004)(26005)(6486002)(36756003)(478600001)(2700100001);DIR:OUT;SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2023 07:39:55.6177 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d55596e5-ae35-4273-c7e7-08dbde9b922f X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AMS1EPF00000040.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB9946 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,FORGED_SPF_HELO,GIT_PATCH_0,KAM_DMARC_NONE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --lnLy9OEdlB9dFHYy Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Hi All, This implements vectorable_early_exit which is used as the codegen part of vectorizing a gcond. For the most part it shares the majority of the code with vectorizable_comparison with addition that it needs to be able to reduce multiple resulting statements into a single one for use in the gcond, and also needs to be able to perform masking on the comparisons. Bootstrapped Regtested on aarch64-none-linux-gnu and no issues. Ok for master? Thanks, Tamar gcc/ChangeLog: * tree-vect-stmts.cc (vectorizable_comparison_1): Support stmts without lhs. (vectorizable_early_exit): New. (vect_analyze_stmt, vect_transform_stmt): Use it. (vect_is_simple_use, vect_get_vector_types_for_stmt): Support gcond. --- inline copy of patch -- diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 36aeca60a22cfaea8d3b43348000d75de1d525c7..4809b822632279493a843d402a833c9267bb315e 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -12475,7 +12475,7 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, vec vec_oprnds0 = vNULL; vec vec_oprnds1 = vNULL; tree mask_type; - tree mask; + tree mask = NULL_TREE; if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo) return false; @@ -12615,8 +12615,9 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, /* Transform. */ /* Handle def. */ - lhs = gimple_assign_lhs (STMT_VINFO_STMT (stmt_info)); - mask = vect_create_destination_var (lhs, mask_type); + lhs = gimple_get_lhs (STMT_VINFO_STMT (stmt_info)); + if (lhs) + mask = vect_create_destination_var (lhs, mask_type); vect_get_vec_defs (vinfo, stmt_info, slp_node, ncopies, rhs1, &vec_oprnds0, vectype, @@ -12630,7 +12631,10 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, gimple *new_stmt; vec_rhs2 = vec_oprnds1[i]; - new_temp = make_ssa_name (mask); + if (lhs) + new_temp = make_ssa_name (mask); + else + new_temp = make_temp_ssa_name (mask_type, NULL, "cmp"); if (bitop1 == NOP_EXPR) { new_stmt = gimple_build_assign (new_temp, code, @@ -12709,6 +12713,196 @@ vectorizable_comparison (vec_info *vinfo, return true; } +/* Check to see if the current early break given in STMT_INFO is valid for + vectorization. */ + +static bool +vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info, + gimple_stmt_iterator *gsi, gimple **vec_stmt, + slp_tree slp_node, stmt_vector_for_cost *cost_vec) +{ + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (!loop_vinfo + || !is_a (STMT_VINFO_STMT (stmt_info))) + return false; + + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_early_exit_def) + return false; + + if (!STMT_VINFO_RELEVANT_P (stmt_info)) + return false; + + gimple_match_op op; + if (!gimple_extract_op (stmt_info->stmt, &op)) + gcc_unreachable (); + gcc_assert (op.code.is_tree_code ()); + auto code = tree_code (op.code); + + tree vectype_out = STMT_VINFO_VECTYPE (stmt_info); + gcc_assert (vectype_out); + + tree var_op = op.ops[0]; + + /* When vectorizing things like pointer comparisons we will assume that + the VF of both operands are the same. e.g. a pointer must be compared + to a pointer. We'll leave this up to vectorizable_comparison_1 to + check further. */ + tree vectype_op = vectype_out; + if (SSA_VAR_P (var_op)) + { + stmt_vec_info operand0_info + = loop_vinfo->lookup_stmt (SSA_NAME_DEF_STMT (var_op)); + if (!operand0_info) + return false; + + /* If we're in a pattern get the type of the original statement. */ + if (STMT_VINFO_IN_PATTERN_P (operand0_info)) + operand0_info = STMT_VINFO_RELATED_STMT (operand0_info); + vectype_op = STMT_VINFO_VECTYPE (operand0_info); + } + + tree truth_type = truth_type_for (vectype_op); + machine_mode mode = TYPE_MODE (truth_type); + int ncopies; + + if (slp_node) + ncopies = 1; + else + ncopies = vect_get_num_copies (loop_vinfo, truth_type); + + vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo); + bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); + + /* Analyze only. */ + if (!vec_stmt) + { + if (direct_optab_handler (cbranch_optab, mode) == CODE_FOR_nothing) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "can't vectorize early exit because the " + "target doesn't support flag setting vector " + "comparisons.\n"); + return false; + } + + if (!expand_vec_cmp_expr_p (vectype_op, truth_type, NE_EXPR)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "can't vectorize early exit because the " + "target does not support boolean vector " + "comparisons for type %T.\n", truth_type); + return false; + } + + if (ncopies > 1 + && direct_optab_handler (ior_optab, mode) == CODE_FOR_nothing) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "can't vectorize early exit because the " + "target does not support boolean vector OR for " + "type %T.\n", truth_type); + return false; + } + + if (!vectorizable_comparison_1 (vinfo, truth_type, stmt_info, code, gsi, + vec_stmt, slp_node, cost_vec)) + return false; + + if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)) + vect_record_loop_mask (loop_vinfo, masks, ncopies, truth_type, NULL); + + return true; + } + + /* Tranform. */ + + tree new_temp = NULL_TREE; + gimple *new_stmt = NULL; + + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "transform early-exit.\n"); + + if (!vectorizable_comparison_1 (vinfo, truth_type, stmt_info, code, gsi, + vec_stmt, slp_node, cost_vec)) + gcc_unreachable (); + + gimple *stmt = STMT_VINFO_STMT (stmt_info); + basic_block cond_bb = gimple_bb (stmt); + gimple_stmt_iterator cond_gsi = gsi_last_bb (cond_bb); + + vec stmts; + + if (slp_node) + stmts = SLP_TREE_VEC_DEFS (slp_node); + else + { + auto vec_stmts = STMT_VINFO_VEC_STMTS (stmt_info); + stmts.create (vec_stmts.length ()); + for (auto stmt : vec_stmts) + stmts.quick_push (gimple_assign_lhs (stmt)); + } + + /* Determine if we need to reduce the final value. */ + if (stmts.length () > 1) + { + /* We build the reductions in a way to maintain as much parallelism as + possible. */ + auto_vec workset (stmts.length ()); + workset.splice (stmts); + while (workset.length () > 1) + { + new_temp = make_temp_ssa_name (truth_type, NULL, "vexit_reduc"); + tree arg0 = workset.pop (); + tree arg1 = workset.pop (); + new_stmt = gimple_build_assign (new_temp, BIT_IOR_EXPR, arg0, arg1); + vect_finish_stmt_generation (loop_vinfo, stmt_info, new_stmt, + &cond_gsi); + if (slp_node) + slp_node->push_vec_def (new_stmt); + else + STMT_VINFO_VEC_STMTS (stmt_info).safe_push (new_stmt); + workset.quick_insert (0, new_temp); + } + } + else + new_temp = stmts[0]; + + gcc_assert (new_temp); + + tree cond = new_temp; + if (masked_loop_p) + { + tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks, ncopies, truth_type, 0); + cond = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask, cond, + &cond_gsi); + } + + /* Now build the new conditional. Pattern gimple_conds get dropped during + codegen so we must replace the original insn. */ + if (is_pattern_stmt_p (stmt_info)) + stmt = STMT_VINFO_STMT (STMT_VINFO_RELATED_STMT (stmt_info)); + + tree t = fold_build2 (NE_EXPR, boolean_type_node, cond, + build_zero_cst (truth_type)); + t = canonicalize_cond_expr_cond (t); + gimple_cond_set_condition_from_tree ((gcond*)stmt, t); + update_stmt (stmt); + + if (slp_node) + slp_node->push_vec_def (stmt); + else + STMT_VINFO_VEC_STMTS (stmt_info).safe_push (stmt); + + + if (!slp_node) + *vec_stmt = stmt; + + return true; +} + /* If SLP_NODE is nonnull, return true if vectorizable_live_operation can handle all live statements in the node. Otherwise return true if STMT_INFO is not live or if vectorizable_live_operation can handle it. @@ -12928,7 +13122,9 @@ vect_analyze_stmt (vec_info *vinfo, || vectorizable_lc_phi (as_a (vinfo), stmt_info, NULL, node) || vectorizable_recurr (as_a (vinfo), - stmt_info, NULL, node, cost_vec)); + stmt_info, NULL, node, cost_vec) + || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node, + cost_vec)); else { if (bb_vinfo) @@ -12951,7 +13147,10 @@ vect_analyze_stmt (vec_info *vinfo, NULL, NULL, node, cost_vec) || vectorizable_comparison (vinfo, stmt_info, NULL, NULL, node, cost_vec) - || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec)); + || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec) + || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node, + cost_vec)); + } if (node) @@ -13110,6 +13309,12 @@ vect_transform_stmt (vec_info *vinfo, gcc_assert (done); break; + case loop_exit_ctrl_vec_info_type: + done = vectorizable_early_exit (vinfo, stmt_info, gsi, &vec_stmt, + slp_node, NULL); + gcc_assert (done); + break; + default: if (!STMT_VINFO_LIVE_P (stmt_info)) { @@ -13511,7 +13716,7 @@ vect_is_simple_use (tree operand, vec_info *vinfo, enum vect_def_type *dt, case vect_first_order_recurrence: dump_printf (MSG_NOTE, "first order recurrence\n"); break; - case vect_early_exit_def: + case vect_early_exit_def: dump_printf (MSG_NOTE, "early exit\n"); break; case vect_unknown_def_type: -- --lnLy9OEdlB9dFHYy Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="rb17969.patch" diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 36aeca60a22cfaea8d3b43348000d75de1d525c7..4809b822632279493a843d402a833c9267bb315e 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -12475,7 +12475,7 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, vec vec_oprnds0 = vNULL; vec vec_oprnds1 = vNULL; tree mask_type; - tree mask; + tree mask = NULL_TREE; if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo) return false; @@ -12615,8 +12615,9 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, /* Transform. */ /* Handle def. */ - lhs = gimple_assign_lhs (STMT_VINFO_STMT (stmt_info)); - mask = vect_create_destination_var (lhs, mask_type); + lhs = gimple_get_lhs (STMT_VINFO_STMT (stmt_info)); + if (lhs) + mask = vect_create_destination_var (lhs, mask_type); vect_get_vec_defs (vinfo, stmt_info, slp_node, ncopies, rhs1, &vec_oprnds0, vectype, @@ -12630,7 +12631,10 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, gimple *new_stmt; vec_rhs2 = vec_oprnds1[i]; - new_temp = make_ssa_name (mask); + if (lhs) + new_temp = make_ssa_name (mask); + else + new_temp = make_temp_ssa_name (mask_type, NULL, "cmp"); if (bitop1 == NOP_EXPR) { new_stmt = gimple_build_assign (new_temp, code, @@ -12709,6 +12713,196 @@ vectorizable_comparison (vec_info *vinfo, return true; } +/* Check to see if the current early break given in STMT_INFO is valid for + vectorization. */ + +static bool +vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info, + gimple_stmt_iterator *gsi, gimple **vec_stmt, + slp_tree slp_node, stmt_vector_for_cost *cost_vec) +{ + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (!loop_vinfo + || !is_a (STMT_VINFO_STMT (stmt_info))) + return false; + + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_early_exit_def) + return false; + + if (!STMT_VINFO_RELEVANT_P (stmt_info)) + return false; + + gimple_match_op op; + if (!gimple_extract_op (stmt_info->stmt, &op)) + gcc_unreachable (); + gcc_assert (op.code.is_tree_code ()); + auto code = tree_code (op.code); + + tree vectype_out = STMT_VINFO_VECTYPE (stmt_info); + gcc_assert (vectype_out); + + tree var_op = op.ops[0]; + + /* When vectorizing things like pointer comparisons we will assume that + the VF of both operands are the same. e.g. a pointer must be compared + to a pointer. We'll leave this up to vectorizable_comparison_1 to + check further. */ + tree vectype_op = vectype_out; + if (SSA_VAR_P (var_op)) + { + stmt_vec_info operand0_info + = loop_vinfo->lookup_stmt (SSA_NAME_DEF_STMT (var_op)); + if (!operand0_info) + return false; + + /* If we're in a pattern get the type of the original statement. */ + if (STMT_VINFO_IN_PATTERN_P (operand0_info)) + operand0_info = STMT_VINFO_RELATED_STMT (operand0_info); + vectype_op = STMT_VINFO_VECTYPE (operand0_info); + } + + tree truth_type = truth_type_for (vectype_op); + machine_mode mode = TYPE_MODE (truth_type); + int ncopies; + + if (slp_node) + ncopies = 1; + else + ncopies = vect_get_num_copies (loop_vinfo, truth_type); + + vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo); + bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); + + /* Analyze only. */ + if (!vec_stmt) + { + if (direct_optab_handler (cbranch_optab, mode) == CODE_FOR_nothing) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "can't vectorize early exit because the " + "target doesn't support flag setting vector " + "comparisons.\n"); + return false; + } + + if (!expand_vec_cmp_expr_p (vectype_op, truth_type, NE_EXPR)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "can't vectorize early exit because the " + "target does not support boolean vector " + "comparisons for type %T.\n", truth_type); + return false; + } + + if (ncopies > 1 + && direct_optab_handler (ior_optab, mode) == CODE_FOR_nothing) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "can't vectorize early exit because the " + "target does not support boolean vector OR for " + "type %T.\n", truth_type); + return false; + } + + if (!vectorizable_comparison_1 (vinfo, truth_type, stmt_info, code, gsi, + vec_stmt, slp_node, cost_vec)) + return false; + + if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)) + vect_record_loop_mask (loop_vinfo, masks, ncopies, truth_type, NULL); + + return true; + } + + /* Tranform. */ + + tree new_temp = NULL_TREE; + gimple *new_stmt = NULL; + + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "transform early-exit.\n"); + + if (!vectorizable_comparison_1 (vinfo, truth_type, stmt_info, code, gsi, + vec_stmt, slp_node, cost_vec)) + gcc_unreachable (); + + gimple *stmt = STMT_VINFO_STMT (stmt_info); + basic_block cond_bb = gimple_bb (stmt); + gimple_stmt_iterator cond_gsi = gsi_last_bb (cond_bb); + + vec stmts; + + if (slp_node) + stmts = SLP_TREE_VEC_DEFS (slp_node); + else + { + auto vec_stmts = STMT_VINFO_VEC_STMTS (stmt_info); + stmts.create (vec_stmts.length ()); + for (auto stmt : vec_stmts) + stmts.quick_push (gimple_assign_lhs (stmt)); + } + + /* Determine if we need to reduce the final value. */ + if (stmts.length () > 1) + { + /* We build the reductions in a way to maintain as much parallelism as + possible. */ + auto_vec workset (stmts.length ()); + workset.splice (stmts); + while (workset.length () > 1) + { + new_temp = make_temp_ssa_name (truth_type, NULL, "vexit_reduc"); + tree arg0 = workset.pop (); + tree arg1 = workset.pop (); + new_stmt = gimple_build_assign (new_temp, BIT_IOR_EXPR, arg0, arg1); + vect_finish_stmt_generation (loop_vinfo, stmt_info, new_stmt, + &cond_gsi); + if (slp_node) + slp_node->push_vec_def (new_stmt); + else + STMT_VINFO_VEC_STMTS (stmt_info).safe_push (new_stmt); + workset.quick_insert (0, new_temp); + } + } + else + new_temp = stmts[0]; + + gcc_assert (new_temp); + + tree cond = new_temp; + if (masked_loop_p) + { + tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks, ncopies, truth_type, 0); + cond = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask, cond, + &cond_gsi); + } + + /* Now build the new conditional. Pattern gimple_conds get dropped during + codegen so we must replace the original insn. */ + if (is_pattern_stmt_p (stmt_info)) + stmt = STMT_VINFO_STMT (STMT_VINFO_RELATED_STMT (stmt_info)); + + tree t = fold_build2 (NE_EXPR, boolean_type_node, cond, + build_zero_cst (truth_type)); + t = canonicalize_cond_expr_cond (t); + gimple_cond_set_condition_from_tree ((gcond*)stmt, t); + update_stmt (stmt); + + if (slp_node) + slp_node->push_vec_def (stmt); + else + STMT_VINFO_VEC_STMTS (stmt_info).safe_push (stmt); + + + if (!slp_node) + *vec_stmt = stmt; + + return true; +} + /* If SLP_NODE is nonnull, return true if vectorizable_live_operation can handle all live statements in the node. Otherwise return true if STMT_INFO is not live or if vectorizable_live_operation can handle it. @@ -12928,7 +13122,9 @@ vect_analyze_stmt (vec_info *vinfo, || vectorizable_lc_phi (as_a (vinfo), stmt_info, NULL, node) || vectorizable_recurr (as_a (vinfo), - stmt_info, NULL, node, cost_vec)); + stmt_info, NULL, node, cost_vec) + || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node, + cost_vec)); else { if (bb_vinfo) @@ -12951,7 +13147,10 @@ vect_analyze_stmt (vec_info *vinfo, NULL, NULL, node, cost_vec) || vectorizable_comparison (vinfo, stmt_info, NULL, NULL, node, cost_vec) - || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec)); + || vectorizable_phi (vinfo, stmt_info, NULL, node, cost_vec) + || vectorizable_early_exit (vinfo, stmt_info, NULL, NULL, node, + cost_vec)); + } if (node) @@ -13110,6 +13309,12 @@ vect_transform_stmt (vec_info *vinfo, gcc_assert (done); break; + case loop_exit_ctrl_vec_info_type: + done = vectorizable_early_exit (vinfo, stmt_info, gsi, &vec_stmt, + slp_node, NULL); + gcc_assert (done); + break; + default: if (!STMT_VINFO_LIVE_P (stmt_info)) { @@ -13511,7 +13716,7 @@ vect_is_simple_use (tree operand, vec_info *vinfo, enum vect_def_type *dt, case vect_first_order_recurrence: dump_printf (MSG_NOTE, "first order recurrence\n"); break; - case vect_early_exit_def: + case vect_early_exit_def: dump_printf (MSG_NOTE, "early exit\n"); break; case vect_unknown_def_type: --lnLy9OEdlB9dFHYy--