From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04on2040.outbound.protection.outlook.com [40.107.7.40]) by sourceware.org (Postfix) with ESMTPS id ADA993857717 for ; Mon, 6 Nov 2023 07:40:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org ADA993857717 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org ADA993857717 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=40.107.7.40 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256432; cv=pass; b=tL5vyPrI+hx7Owmeuz3VZ0cYsiWonWfbjTnHkHWbF4Gx2PrMxhzrIbq+w93TMtDSyd9xg99OOA8ziCEBdbV9l0dvzr/+ppQwmKUpbp26xyToU6AG5jGjztbsalJdgcXxb1OkXZrhiavb4xonI9QQC/DQ1x62kuhJAn39Rw43gBQ= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1699256432; c=relaxed/simple; bh=6tbshpWQ9g1hOw1XfdAGQwTZBfMbIWx65UsNDKXH00c=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=Isp8ajyepjqtbJm/NQerlO27XBneSH5vzLc6tirPEQO5VO4tCzqckDKKKAUJkOn3vL9H3uWJt8VGz04RSJQU4JpoqbuxVmU/3hHrlT1ATb2SK3CnbMmM5NNe5hAAUJqAOQi48tT9RxrhNu7zJTIdSdhifq7JOOITHfXtvVLz2Ig= ARC-Authentication-Results: i=3; server2.sourceware.org ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=lAb7oMURnMdrhWwQKHJIoti/fxbPXYFXTV0460FhjkSDjSz8r3UgllWBDqQAppNR+d2qvAez1v0GlWQTDt17NXtmIFF5RrKxJHJPfBdHzLw+dz/2VLb5GEi7oJHsK6OYBfnQmAHcoNKMwk1E5ZlQYHq6B6YGjvQyf7fV+etqHbcGgJe1BUbn+zEBOJpqGSxkAxBYnZ3h6/hWMywwhqXhEhkxRTfksQGeM106xZIJtmFmzNufPlbRYftX7ZqTrX3Fh2Q7Ra2U5KBRXE8/KkDiA//Bebu/R7klEtBFlYhQbJ0GU8fe0DQHo4y5whtRV0dodSqp26+AMYhLNHGLlLx/mQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=b/KvuCA7z0bGhNU230+Vn8o3aVtfmZWUwRWFDZOm8QB+WnYuLk+YMA8zReJeTgDSe3fareQowFUI5DmilPRaZyl3se5va+gGBbwOxHGxB9BXggqfGqTkEjAVtiQdH7MaEcWmFjosstDXHNLEt6abhZeFw/Mg0h5x9RKDHabIhzO1jR5cDFSFSDZ5zcPeiNhTHt8ddVSzq9ULMdrQYncWA9DrQIJzMWkP0Ap62w1fju/MKfR7saeQnZccO5e+AONdXamfjh+LbR6OU4oFm15JtVC9epjybXPb96/VLl8sY2agUUpld/1uTc/E6QUOv5qsJ3FyMD9N4ZkfrPwnKRunPA== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dkim=[1,1,header.d=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=zGLECUtIty2hs+mEevNFwiW+kmaOvBnUSFrlQQHNiH1lmFFOaSST0tY63J4v1CH3eyywN6jhB5Vw4IJaHsiGTEAyUyS45WGq49I65I3hXlw45lbws6f8kVUzYjVAnss+dtjWjNtYqL9hRZQlDEvJTONtCzmyACpwTbIxcW5DFFo= Received: from DUZPR01CA0065.eurprd01.prod.exchangelabs.com (2603:10a6:10:3c2::8) by AS4PR08MB7735.eurprd08.prod.outlook.com (2603:10a6:20b:512::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.27; Mon, 6 Nov 2023 07:40:26 +0000 Received: from DU6PEPF0000B621.eurprd02.prod.outlook.com (2603:10a6:10:3c2:cafe::d3) by DUZPR01CA0065.outlook.office365.com (2603:10a6:10:3c2::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28 via Frontend Transport; Mon, 6 Nov 2023 07:40:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DU6PEPF0000B621.mail.protection.outlook.com (10.167.8.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6977.16 via Frontend Transport; Mon, 6 Nov 2023 07:40:25 +0000 Received: ("Tessian outbound 20615a7e7970:v228"); Mon, 06 Nov 2023 07:40:25 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 7b30dd7d4c61451e X-CR-MTA-TID: 64aa7808 Received: from 214080e359b5.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 857409DD-D098-4D01-91D8-82547B30D564.1; Mon, 06 Nov 2023 07:40:19 +0000 Received: from EUR04-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 214080e359b5.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 06 Nov 2023 07:40:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E8D+LeIIp4N1UzZEdfRjLAjni58avbuDE/KgmvHTJH37V6TxMVXSrGa5TgBBDoL7M1iybCi3JMc3wvg4wCCoq4rSlv9Uub8SNtNAqEIrLo6yXy+m7IQ5l9vhmqTNUGQC7woeYwcqlgoSMkYl0zOr8J5fVm/vGwv+JP2JkDiUfOyctpRhBRSTAYMe0J702ElWt29BV2E+/1PnnPBgaJUxawUSO7wcEi/uOj6XWWZ3UpopS2vPb2x8Whlrp5Wtl7x04Q1yqU70EQ6W5M4zdS3EMQbWuwFYIWlziptYUrylSwFypVDao7/JygFx5u4b2ZppH8MSn8VqzAk3QgqzVfsMcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=ihlIhQjh3vscpm+cvYHcwboywvNK890zfMumiuJLo7HjTODnsMmAaxGfLbkvOqGaAZbsAW/7JRpcyztVeLoXlRwGF2IwMAhAaAsI3IdfThLhIJEVARf8KbdV3Belmmyw5Vp7gcV8fPLJGQOOq/O8+jBdM2MM+TZKremjEDnA4ogFvvk525E7RcxMW7a4g2Mu2kYo7c/QT02kka39AHBoCeCBTBCMI3jh3fB9yHeGxkx8+0Ib4oRkc3KA96JeZuz0L+iXGZ5gSdXZW38wbUp6jGNE8P2iDYc6iZC8qZXbJUUxDGgzX5E22i0T1wd5y2BiaxkYLYRD6U21qFnIHBoZww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qxl83AR98eelgy8HBXksJKU0rKBbNx2ShIdYqtDbLzg=; b=zGLECUtIty2hs+mEevNFwiW+kmaOvBnUSFrlQQHNiH1lmFFOaSST0tY63J4v1CH3eyywN6jhB5Vw4IJaHsiGTEAyUyS45WGq49I65I3hXlw45lbws6f8kVUzYjVAnss+dtjWjNtYqL9hRZQlDEvJTONtCzmyACpwTbIxcW5DFFo= Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; Received: from VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) by DBBPR08MB10676.eurprd08.prod.outlook.com (2603:10a6:10:530::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.28; Mon, 6 Nov 2023 07:40:16 +0000 Received: from VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0]) by VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::26aa:efdd:a74a:27d0%5]) with mapi id 15.20.6954.028; Mon, 6 Nov 2023 07:40:16 +0000 Date: Mon, 6 Nov 2023 07:40:13 +0000 From: Tamar Christina To: gcc-patches@gcc.gnu.org Cc: nd@arm.com, rguenther@suse.de, jlaw@ventanamicro.com Subject: [PATCH 11/21]middle-end: wire through peeling changes and dominator updates after guard edge split Message-ID: Content-Type: multipart/mixed; boundary="s7mzVILw8lrMQknv" Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: LO3P123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:ba::20) To VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) MIME-Version: 1.0 X-MS-TrafficTypeDiagnostic: VI1PR08MB5325:EE_|DBBPR08MB10676:EE_|DU6PEPF0000B621:EE_|AS4PR08MB7735:EE_ X-MS-Office365-Filtering-Correlation-Id: 77c12a0d-4881-47d1-c871-08dbde9ba42d x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: U65UqWvmjk5wVZs9O3EwjpfBjxEqzBBXMKyZylBUJbm4xmtGl3Yt1XtkOdrsP6hchChJj/dcnf2QzXaqeuD21BCppIRDhjtxL0ouuvZpdr4NHGCxsV7mPn7R1+XsJ7Yy+CpnjtPOB2V705RtoEHgG7aX35nLTJQNwlcRWpEfFeKyB7LeulV3tOosd90kGCFy9qM8RJ+nNW6piMRnL0Z4tLbK00uuDFiLanSlzUx4cwe4jZgDHx3brFvRlx5/Y0x5b8eDGDL1crlsPTld7kCaZ/m7I2IffSMnX1E8qm6LqMCuRNpm3Ebfk3Z0dwGWJh9ya0AB/ANSPmMGJzqUllCx5sxixQy3RcjC74K4SryAtEtQs91l6p1pNMYC2O2tTgZi6NhVasYoPofKW/1jd3B9blokdvA8trUz5mNV4tfKsMgY7zLXLkT1sSEOSz2+Mlxxj4qrnysWyS+EPn7169P1RLhNNbjGCAgokRHT+dlfwcf0Uf8G5+RG1dUPHvWfMK8MD5JDIwZw/6vF7Mn+OU5Ykz7Eu4l0KS2tVrBAM5e/aup+l5Pv65bthYNAJjslfcitCylgiV/GI/BKHMGDSqbROyBfETO5VcUb3Dlrj3pwTTQuWXPEvN7XtB86zF7nWJbY X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR08MB5325.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(396003)(376002)(366004)(346002)(39860400002)(230922051799003)(186009)(64100799003)(1800799009)(451199024)(6916009)(4326008)(8676002)(8936002)(41300700001)(36756003)(2906002)(235185007)(38100700002)(5660300002)(6486002)(66556008)(6512007)(316002)(66476007)(86362001)(44832011)(15650500001)(4743002)(66946007)(83380400001)(26005)(2616005)(6666004)(6506007)(478600001)(33964004)(44144004)(2700100001);DIR:OUT;SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB10676 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DU6PEPF0000B621.eurprd02.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: eaffacca-f5fc-4df7-ada2-08dbde9b9eaf X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fdvc69u07+r2Wy+H2kXc3yCa1rDvGjw4rmlXZ8Z9Vg7f7fZM5/PEKsxHg5fCq1An5vntxDp43FLeTZjx4QIHi536BIU23TdEB2TlcQQHm6SrawPaeouRbhaaSfdwF4sV5NAzsVCrLmCNFeEXXPw0c/3VJKHhwqYCSGaM5WvlaHT+qQY0wuSXt7SAOIYP9x8mhf3hcY60ZTZIQxJl1EN6tqamJfyo/h92Ns22Y+7oCG3QCNCgMu/gipmSl5J6z1fTvIqihShPNMUsLVIjsgCdPkQH1o7jaM/GQotxV3RT9a1kVFExnc3jlEpYiyhQQS+j2Xnub79P0PmDkprHc+z4lSI5foUX7OAb9BAAcdVjP6sQcApljNVrQfJRnKaF1EmqL7s2gBt4O1m/R6rlxYO5r9/nQMy7CqFk/nJSKaF2OY7/t3HxkpLNxGzeX0cePyhCDVdlFf32s+7B1URMKFVmy4naMsRj+oaPJb8JcTw4Gpe3Fnwg5v85U+RxQpXxV+kiNm/vVrkFTcAisWgms5vbLkGLfCkyEMgR/ksiy1IZfqh4lykQ0gZFQhDBcIQLjwT3hAUFX/ztZEWP8Cmf8EiXm3PyOQWQ9+LQte9OT+4kRqerOMYQGi2l1t+OQCBZocaR7blOBGSUV8zbUQQNV+Z1h9/REXxHS0wGUbAUJ980IMOfkemzpWRWMRAQZtv+1y9SNyti42p1XFOqpPOO3vk7ztQuauxLGwYIjlLnr2EC/b8/4l64hKqmHEkzZXHyJsV/wabQ2PeCdW19057JOSzU2g== X-Forefront-Antispam-Report: CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(376002)(39860400002)(346002)(396003)(230922051799003)(82310400011)(186009)(1800799009)(451199024)(64100799003)(40470700004)(36840700001)(46966006)(33964004)(107886003)(2616005)(6666004)(4743002)(336012)(44144004)(6506007)(6512007)(41300700001)(26005)(47076005)(83380400001)(8936002)(4326008)(8676002)(15650500001)(2906002)(36860700001)(44832011)(5660300002)(235185007)(6486002)(6916009)(478600001)(316002)(70586007)(70206006)(356005)(81166007)(82740400003)(36756003)(86362001)(40460700003)(40480700001)(2700100001);DIR:OUT;SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Nov 2023 07:40:25.8818 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 77c12a0d-4881-47d1-c871-08dbde9ba42d X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DU6PEPF0000B621.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR08MB7735 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,FORGED_SPF_HELO,GIT_PATCH_0,KAM_DMARC_NONE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: --s7mzVILw8lrMQknv Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Hi All, This wires through the final bits to support adding the guard block between the loop and epilog. For an "inverted loop", i.e. one where an early exit was chosen as the main exit then we can never skip the scalar loop since we know we have side effects to still perform. For those cases we always restart the scalar loop regardless of which exit is taken. When we add the guard we also need to update the dominators calculated by the peeling code as the bypass edge changes the dominators. Bootstrapped Regtested on aarch64-none-linux-gnu and no issues. Ok for master? Thanks, Tamar gcc/ChangeLog: * tree-vect-loop-manip.cc (vect_do_peeling): --- inline copy of patch -- diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index abd905b78f3661f80168c3866d7c3e68a9c15521..eef2bb50c1505f5cf802d5d80300affc2cbe69f6 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -3512,11 +3512,14 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, epilog = vect_epilogues ? get_loop_copy (loop) : scalar_loop; edge epilog_e = vect_epilogues ? e : scalar_e; edge new_epilog_e = NULL; - epilog = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, - epilog_e, e, - &new_epilog_e); + auto_vec doms; + epilog + = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, epilog_e, e, + &new_epilog_e, true, &doms); + LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo) = new_epilog_e; gcc_assert (epilog); + gcc_assert (new_epilog_e); epilog->force_vectorize = false; bb_before_epilog = loop_preheader_edge (epilog)->src; @@ -3610,10 +3613,16 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, } - if (skip_epilog) + if (skip_epilog || LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) { - guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, + /* For the case where a different exit was chosen we must execute + the scalar loop with the remaining iterations. */ + if (inversed_iv) + guard_cond = boolean_false_node; + else + guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, niters, niters_vector_mult_vf); + guard_bb = LOOP_VINFO_IV_EXIT (loop_vinfo)->dest; edge epilog_e = LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo); guard_to = split_edge (epilog_e); @@ -3621,11 +3630,13 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, skip_vector ? anchor : guard_bb, prob_epilog.invert (), irred_flag); + doms.safe_push (guard_to); if (vect_epilogues) epilogue_vinfo->skip_this_loop_edge = guard_e; edge main_iv = LOOP_VINFO_IV_EXIT (loop_vinfo); slpeel_update_phi_nodes_for_guard2 (loop, epilog, main_iv, guard_e, epilog_e); + /* Only need to handle basic block before epilog loop if it's not the guard_bb, which is the case when skip_vector is true. */ if (guard_bb != bb_before_epilog) @@ -3637,6 +3648,10 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, scale_loop_profile (epilog, prob_epilog, -1); } + /* Recalculate the dominators after adding the guard edge. */ + if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) + iterate_fix_dominators (CDI_DOMINATORS, doms, false); + unsigned HOST_WIDE_INT bound; if (bound_scalar.is_constant (&bound)) { -- --s7mzVILw8lrMQknv Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="rb17971.patch" diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index abd905b78f3661f80168c3866d7c3e68a9c15521..eef2bb50c1505f5cf802d5d80300affc2cbe69f6 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -3512,11 +3512,14 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, epilog = vect_epilogues ? get_loop_copy (loop) : scalar_loop; edge epilog_e = vect_epilogues ? e : scalar_e; edge new_epilog_e = NULL; - epilog = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, - epilog_e, e, - &new_epilog_e); + auto_vec doms; + epilog + = slpeel_tree_duplicate_loop_to_edge_cfg (loop, e, epilog, epilog_e, e, + &new_epilog_e, true, &doms); + LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo) = new_epilog_e; gcc_assert (epilog); + gcc_assert (new_epilog_e); epilog->force_vectorize = false; bb_before_epilog = loop_preheader_edge (epilog)->src; @@ -3610,10 +3613,16 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, } - if (skip_epilog) + if (skip_epilog || LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) { - guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, + /* For the case where a different exit was chosen we must execute + the scalar loop with the remaining iterations. */ + if (inversed_iv) + guard_cond = boolean_false_node; + else + guard_cond = fold_build2 (EQ_EXPR, boolean_type_node, niters, niters_vector_mult_vf); + guard_bb = LOOP_VINFO_IV_EXIT (loop_vinfo)->dest; edge epilog_e = LOOP_VINFO_EPILOGUE_IV_EXIT (loop_vinfo); guard_to = split_edge (epilog_e); @@ -3621,11 +3630,13 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, skip_vector ? anchor : guard_bb, prob_epilog.invert (), irred_flag); + doms.safe_push (guard_to); if (vect_epilogues) epilogue_vinfo->skip_this_loop_edge = guard_e; edge main_iv = LOOP_VINFO_IV_EXIT (loop_vinfo); slpeel_update_phi_nodes_for_guard2 (loop, epilog, main_iv, guard_e, epilog_e); + /* Only need to handle basic block before epilog loop if it's not the guard_bb, which is the case when skip_vector is true. */ if (guard_bb != bb_before_epilog) @@ -3637,6 +3648,10 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1, scale_loop_profile (epilog, prob_epilog, -1); } + /* Recalculate the dominators after adding the guard edge. */ + if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) + iterate_fix_dominators (CDI_DOMINATORS, doms, false); + unsigned HOST_WIDE_INT bound; if (bound_scalar.is_constant (&bound)) { --s7mzVILw8lrMQknv--