From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04on2085.outbound.protection.outlook.com [40.107.7.85]) by sourceware.org (Postfix) with ESMTPS id 0867B3858C5E for ; Thu, 23 Feb 2023 15:11:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0867B3858C5E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=37cNnNgjUJmZ4mIwaE/BTu779JCAMfqDI8giFTnEKwo=; b=WGyXlfngvDZNWjYS2c/tvgCtusstzotyetfyJwkKv3TRhGf3kDXAX3575wqu1qslwx2tObShzVTh4CDvT0nTYTqK2deo9Wn/OedJDPxI9Mq7nQgDmmWBNb3ulEhbSHYGYDBqxVGq14avYQ5sXcWCmjrxMrtK3qxWc77bd/KI6xA= Received: from AM6P195CA0040.EURP195.PROD.OUTLOOK.COM (2603:10a6:209:87::17) by GV2PR08MB9206.eurprd08.prod.outlook.com (2603:10a6:150:d5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.19; Thu, 23 Feb 2023 15:11:49 +0000 Received: from AM7EUR03FT042.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:87:cafe::8) by AM6P195CA0040.outlook.office365.com (2603:10a6:209:87::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.21 via Frontend Transport; Thu, 23 Feb 2023 15:11:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM7EUR03FT042.mail.protection.outlook.com (100.127.140.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.21 via Frontend Transport; Thu, 23 Feb 2023 15:11:49 +0000 Received: ("Tessian outbound 333ca28169fa:v132"); Thu, 23 Feb 2023 15:11:49 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 961ca1fe8b68b062 X-CR-MTA-TID: 64aa7808 Received: from 3db0d2c4c236.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id FA4547F2-0196-4FCA-92AE-CFE3E9747D94.1; Thu, 23 Feb 2023 15:11:42 +0000 Received: from EUR02-VI1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 3db0d2c4c236.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 23 Feb 2023 15:11:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AN0m0TLHPVJ5fV6d8nFlxLFRLrNsBkPlHbIwWYKr5mBgkzJKAhpr64MUbP/HYQO7euGRU6NgE06fmktHcji7T+f43i9L6ZS9O8uHKQnzfN0/0M4y4G2zREAEGJWNZXG6tYq0QWRhB3Gw9SRPBO2t/bGDWtojkop1ppS8mqje2qXZXGbFyD9q/ByFJ5fjxllvOpBPkOreaAsOj5J2RVi6DY3rkUKsQsc+12Ahu4LxeTe48V2WF9eX7rM9omfvXzKEzVRVkHelUXydTvQYcMF/hZ1NNCxaGqHrHiDQIaZYwIFBFjrLFDeJg61DrLRUtugBSlCLHVYoQ+3UAA6QboqUkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=37cNnNgjUJmZ4mIwaE/BTu779JCAMfqDI8giFTnEKwo=; b=IN/H2Q9OdMHMON55JEIpFpt4G/SGjn7nykBmf9v+hA5YjpdLt0G2jikt7URbJVAht3WvE+60dFCeXOp/EwvbWXr1dnaqbLsLcRPR3P0aLQrW1qXViKEYDVx7EqWV5m44M7lG90NC+38819fQdL1DB15B3FKylkMBfAuUdX63cKl8I89GN2Af0k6tFdOUYNozFUEeHFipIMWEuPP7V4Z7qeAo4yuJTq5DvwkBGLG/bqk8KB6MWjhVLEhoXmZmEGiYHhJnAo8AjrRp6YDOB3sURcKJXXPpe4NnzH4lNSQpYRcL2555Ye/Km3hRsvBVwxB02lSowfSlsRhPEPuL9dQ48w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=37cNnNgjUJmZ4mIwaE/BTu779JCAMfqDI8giFTnEKwo=; b=WGyXlfngvDZNWjYS2c/tvgCtusstzotyetfyJwkKv3TRhGf3kDXAX3575wqu1qslwx2tObShzVTh4CDvT0nTYTqK2deo9Wn/OedJDPxI9Mq7nQgDmmWBNb3ulEhbSHYGYDBqxVGq14avYQ5sXcWCmjrxMrtK3qxWc77bd/KI6xA= Received: from PAWPR08MB8982.eurprd08.prod.outlook.com (2603:10a6:102:33f::20) by PAWPR08MB9030.eurprd08.prod.outlook.com (2603:10a6:102:342::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.18; Thu, 23 Feb 2023 15:11:40 +0000 Received: from PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::dc17:8fa2:cce5:3573]) by PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::dc17:8fa2:cce5:3573%8]) with mapi id 15.20.6134.021; Thu, 23 Feb 2023 15:11:40 +0000 From: Wilco Dijkstra To: GCC Patches CC: Richard Sandiford , Kyrylo Tkachov Subject: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891] Thread-Topic: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891] Thread-Index: AQHZR5de3lyELmNywka8opZDiKy6FQ== Date: Thu, 23 Feb 2023 15:11:39 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: PAWPR08MB8982:EE_|PAWPR08MB9030:EE_|AM7EUR03FT042:EE_|GV2PR08MB9206:EE_ X-MS-Office365-Filtering-Correlation-Id: e9780355-8682-4fae-77f0-08db15b04972 x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: zfC2SuR1o//vKpOsW+k8S0MH2AiQsSTh/fegFIk+nChExNmLmAH47amR1eYOV/sN9z7nxdUtMHsiPHsz6ZGaajKxQJbh9QidWbKvqFNQjoZEkcqD34V1uESLZLSMtpaTBuGIK0GUTTPiBI44wBer6qxwmfDrzufk7fdxFJpACb8dKqwSu2PuYcL2wegNhsm5lrdq6+tO9KWlY1fppoHRWZcQmtqw93WO0FfTTSAp0CbK7RWIgJGLZzCBLg590W9H6OHybMuLEjALWhJmBQq/SNOC8AqGUamyzpU+Odoex0VcBT6VwTQ8n15mDbD3iMQVjD3m/v7oTnpreAFBzJ1qpDM9aYA0jVfC55JOfirMBwdBKkf5mm8aT1XFrbaeCBr6qbQUECTzeLCxqkEhzv99rDZzGWR4rwfMmonEb2kdt42npZQ/YHtKKaZ3t+j7MgH1dBET1jFhmwG4Ig75XqqaJFS8twaZA9U3lum7vXiC/cLj33aKYdJCnWs4WTMBAynfnXTBNXz3qUfopc9tKzu6Skrvp4UC0nLbchOuth5NSA3IepHsqlVFwziLd/GgUT5dJjNHuRXCPgOVCruCuYLNkGVDecB83jhmkOWSDz4YnVgFt77BetbssUbnoF/DBDjcWgPzo0F2/icEtEHgmKw5p9z5eXdqEePvvlxUMBkgmogzb7mdpK5owGTI/BH3O7Tq9C0SJlgClpBbEylcy3qPJA== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAWPR08MB8982.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(346002)(376002)(39860400002)(396003)(136003)(366004)(451199018)(26005)(76116006)(41300700001)(66476007)(55016003)(7696005)(9686003)(6506007)(4326008)(66556008)(66446008)(316002)(86362001)(33656002)(52536014)(186003)(8676002)(5660300002)(66946007)(83380400001)(64756008)(91956017)(122000001)(478600001)(54906003)(6916009)(8936002)(71200400001)(38100700002)(38070700005)(2906002);DIR:OUT;SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB9030 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM7EUR03FT042.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: e12d8d18-1207-4e4f-c451-08db15b04399 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RJb9//cz6m9KLXG1eYS9vX2Lv8U4mgR8ZgHsLLqvZ+EZf4NUOfZOknmdp0dFb4EWaepcXWEP6Vq2cb5NTBWExV8xqNXyWwQuH6MpZPwM5HrL73G+fUBoOPgrWgIReqpvtQOdisft7FB4kibO4zErN9u9dHuxLs00ew611G4lwfJQ3q14HmxWiaIdoz2DUMzN4qhmg/9KybB6cv2k8+S71xBap2zRfkir2BbRuFJyYNy4gBIGpYn4YMHFps2GlqSfRRzafFDLSpZ+WC+RXICDrhfFnL9qquVt4bZktkUmRC40byiYyG/S9G1MEgHT/zZQIAfq8FW6SaeGoFlsg0YUPeVh6F0eVXNFMFoBe6sEylHa4ClFFEfLlkr0r6yNxJKDgFKi+YM50hF0TZqmDBYf5qou9+GE66Ukoi4SR1sEmNUiT9nrjV7lk/gH3GL/eZkD6Jh+Vp3Y4s+WE8bXQSdTqHtaXlGq+fE92sez+V7/zGJTqwuF2SdKOCSSLdcEbKasFyNEOtNZht1U5jvYBnmhSTufDLyrUXhh82h5AjzzCmxpAsJ4J698NEGL3tcvUcEHgXjDiwGKRIMko4bIcEb809ewQD2vfSyKDAyUwGAityuapsMWNzkoTDBCoO/PUK6NGcA5ph3Su2syCeOPesQFDg5JU7CVq4g/dA5MOg5wHUx5zsZZ+33pAozK7/Je/CQiULksm+oZBDHC9ypIO4h3Cw== X-Forefront-Antispam-Report: CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230025)(4636009)(376002)(39860400002)(346002)(136003)(396003)(451199018)(46966006)(40470700004)(36840700001)(82310400005)(83380400001)(36860700001)(47076005)(6506007)(7696005)(33656002)(40460700003)(81166007)(54906003)(82740400003)(316002)(186003)(336012)(40480700001)(478600001)(356005)(26005)(9686003)(2906002)(8936002)(52536014)(5660300002)(41300700001)(4326008)(86362001)(70586007)(55016003)(6916009)(70206006)(8676002);DIR:OUT;SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2023 15:11:49.2941 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e9780355-8682-4fae-77f0-08db15b04972 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM7EUR03FT042.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV2PR08MB9206 X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,FORGED_SPF_HELO,GIT_PATCH_0,KAM_DMARC_NONE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SCC_10_SHORT_WORD_LINES,SCC_20_SHORT_WORD_LINES,SCC_5_SHORT_WORD_LINES,SPF_HELO_PASS,SPF_NONE,TXREP,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: =0A= The LSE2 ifunc for 16-byte atomic load requires a barrier before the LDP -= =0A= without it, it effectively has Load-AcquirePC semantics similar to LDAPR,= =0A= which is less restrictive than what __ATOMIC_SEQ_CST requires. This patch= =0A= fixes this and adds comments to make it easier to see which sequence is=0A= used for each case. Use a load/store exclusive loop for store to simplify= =0A= testing memory ordering is correct (it is slightly faster too). =0A= =0A= Passes regress, OK for commit?=0A= =0A= libatomic/=0A= PR libgcc/108891=0A= config/linux/aarch64/atomic_16.S: Fix libat_load_16_i1.=0A= Add comments describing the memory order.=0A= =0A= ---=0A= =0A= diff --git a/libatomic/config/linux/aarch64/atomic_16.S b/libatomic/config/= linux/aarch64/atomic_16.S=0A= index 732c3534a06678664a252bdbc53652eeab0af506..05439ce394b9653c9bcb582761f= f7aaa7c8f9643 100644=0A= --- a/libatomic/config/linux/aarch64/atomic_16.S=0A= +++ b/libatomic/config/linux/aarch64/atomic_16.S=0A= @@ -72,33 +72,38 @@ name: \=0A= =0A= ENTRY (libat_load_16_i1)=0A= cbnz w1, 1f=0A= +=0A= + /* RELAXED. */=0A= ldp res0, res1, [x0]=0A= ret=0A= 1:=0A= - cmp w1, ACQUIRE=0A= - b.hi 2f=0A= + cmp w1, SEQ_CST=0A= + b.eq 2f=0A= +=0A= + /* ACQUIRE/CONSUME (Load-AcquirePC semantics). */=0A= ldp res0, res1, [x0]=0A= dmb ishld=0A= ret=0A= -2:=0A= +=0A= + /* SEQ_CST. */=0A= +2: ldar tmp0, [x0] /* Block reordering with Store-Release instr. */=0A= ldp res0, res1, [x0]=0A= - dmb ish=0A= + dmb ishld=0A= ret=0A= END (libat_load_16_i1)=0A= =0A= =0A= ENTRY (libat_store_16_i1)=0A= cbnz w4, 1f=0A= +=0A= + /* RELAXED. */=0A= stp in0, in1, [x0]=0A= ret=0A= -1:=0A= - dmb ish=0A= - stp in0, in1, [x0]=0A= - cmp w4, SEQ_CST=0A= - beq 2f=0A= - ret=0A= -2:=0A= - dmb ish=0A= +=0A= + /* RELEASE/SEQ_CST. */=0A= +1: ldaxp xzr, tmp0, [x0]=0A= + stlxp w4, in0, in1, [x0]=0A= + cbnz w4, 1b=0A= ret=0A= END (libat_store_16_i1)=0A= =0A= @@ -106,29 +111,33 @@ END (libat_store_16_i1)=0A= ENTRY (libat_exchange_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= stxp w4, in0, in1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= 2:=0A= cmp w4, ACQUIRE=0A= b.hi 4f=0A= -3:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME. */=0A= +3: ldaxp res0, res1, [x5]=0A= stxp w4, in0, in1, [x5]=0A= cbnz w4, 3b=0A= ret=0A= 4:=0A= cmp w4, RELEASE=0A= b.ne 6f=0A= -5:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELEASE. */=0A= +5: ldxp res0, res1, [x5]=0A= stlxp w4, in0, in1, [x5]=0A= cbnz w4, 5b=0A= ret=0A= -6:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQ_REL/SEQ_CST. */=0A= +6: ldaxp res0, res1, [x5]=0A= stlxp w4, in0, in1, [x5]=0A= cbnz w4, 6b=0A= ret=0A= @@ -142,6 +151,8 @@ ENTRY (libat_compare_exchange_16_i1)=0A= cbz w4, 2f=0A= cmp w4, RELEASE=0A= b.hs 3f=0A= +=0A= + /* ACQUIRE/CONSUME. */=0A= caspa exp0, exp1, in0, in1, [x0]=0A= 0:=0A= cmp exp0, tmp0=0A= @@ -153,15 +164,18 @@ ENTRY (libat_compare_exchange_16_i1)=0A= stp exp0, exp1, [x1]=0A= mov x0, 0=0A= ret=0A= -2:=0A= - casp exp0, exp1, in0, in1, [x0]=0A= +=0A= + /* RELAXED. */=0A= +2: casp exp0, exp1, in0, in1, [x0]=0A= b 0b=0A= -3:=0A= - b.hi 4f=0A= +=0A= + /* RELEASE. */=0A= +3: b.hi 4f=0A= caspl exp0, exp1, in0, in1, [x0]=0A= b 0b=0A= -4:=0A= - caspal exp0, exp1, in0, in1, [x0]=0A= +=0A= + /* ACQ_REL/SEQ_CST. */=0A= +4: caspal exp0, exp1, in0, in1, [x0]=0A= b 0b=0A= END (libat_compare_exchange_16_i1)=0A= =0A= @@ -169,15 +183,17 @@ END (libat_compare_exchange_16_i1)=0A= ENTRY (libat_fetch_add_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= adds tmplo, reslo, inlo=0A= adc tmphi, reshi, inhi=0A= stxp w4, tmp0, tmp1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= adds tmplo, reslo, inlo=0A= adc tmphi, reshi, inhi=0A= stlxp w4, tmp0, tmp1, [x5]=0A= @@ -189,15 +205,17 @@ END (libat_fetch_add_16_i1)=0A= ENTRY (libat_add_fetch_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= adds reslo, reslo, inlo=0A= adc reshi, reshi, inhi=0A= stxp w4, res0, res1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= adds reslo, reslo, inlo=0A= adc reshi, reshi, inhi=0A= stlxp w4, res0, res1, [x5]=0A= @@ -209,15 +227,17 @@ END (libat_add_fetch_16_i1)=0A= ENTRY (libat_fetch_sub_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= subs tmplo, reslo, inlo=0A= sbc tmphi, reshi, inhi=0A= stxp w4, tmp0, tmp1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= subs tmplo, reslo, inlo=0A= sbc tmphi, reshi, inhi=0A= stlxp w4, tmp0, tmp1, [x5]=0A= @@ -229,15 +249,17 @@ END (libat_fetch_sub_16_i1)=0A= ENTRY (libat_sub_fetch_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= subs reslo, reslo, inlo=0A= sbc reshi, reshi, inhi=0A= stxp w4, res0, res1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= subs reslo, reslo, inlo=0A= sbc reshi, reshi, inhi=0A= stlxp w4, res0, res1, [x5]=0A= @@ -249,15 +271,17 @@ END (libat_sub_fetch_16_i1)=0A= ENTRY (libat_fetch_or_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= orr tmp0, res0, in0=0A= orr tmp1, res1, in1=0A= stxp w4, tmp0, tmp1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= orr tmp0, res0, in0=0A= orr tmp1, res1, in1=0A= stlxp w4, tmp0, tmp1, [x5]=0A= @@ -269,15 +293,17 @@ END (libat_fetch_or_16_i1)=0A= ENTRY (libat_or_fetch_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= orr res0, res0, in0=0A= orr res1, res1, in1=0A= stxp w4, res0, res1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= orr res0, res0, in0=0A= orr res1, res1, in1=0A= stlxp w4, res0, res1, [x5]=0A= @@ -289,15 +315,17 @@ END (libat_or_fetch_16_i1)=0A= ENTRY (libat_fetch_and_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= and tmp0, res0, in0=0A= and tmp1, res1, in1=0A= stxp w4, tmp0, tmp1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= and tmp0, res0, in0=0A= and tmp1, res1, in1=0A= stlxp w4, tmp0, tmp1, [x5]=0A= @@ -309,15 +337,17 @@ END (libat_fetch_and_16_i1)=0A= ENTRY (libat_and_fetch_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= and res0, res0, in0=0A= and res1, res1, in1=0A= stxp w4, res0, res1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= and res0, res0, in0=0A= and res1, res1, in1=0A= stlxp w4, res0, res1, [x5]=0A= @@ -329,15 +359,17 @@ END (libat_and_fetch_16_i1)=0A= ENTRY (libat_fetch_xor_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= eor tmp0, res0, in0=0A= eor tmp1, res1, in1=0A= stxp w4, tmp0, tmp1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= eor tmp0, res0, in0=0A= eor tmp1, res1, in1=0A= stlxp w4, tmp0, tmp1, [x5]=0A= @@ -349,15 +381,17 @@ END (libat_fetch_xor_16_i1)=0A= ENTRY (libat_xor_fetch_16_i1)=0A= mov x5, x0=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= eor res0, res0, in0=0A= eor res1, res1, in1=0A= stxp w4, res0, res1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= eor res0, res0, in0=0A= eor res1, res1, in1=0A= stlxp w4, res0, res1, [x5]=0A= @@ -371,15 +405,17 @@ ENTRY (libat_fetch_nand_16_i1)=0A= mvn in0, in0=0A= mvn in1, in1=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= orn tmp0, in0, res0=0A= orn tmp1, in1, res1=0A= stxp w4, tmp0, tmp1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= orn tmp0, in0, res0=0A= orn tmp1, in1, res1=0A= stlxp w4, tmp0, tmp1, [x5]=0A= @@ -393,15 +429,17 @@ ENTRY (libat_nand_fetch_16_i1)=0A= mvn in0, in0=0A= mvn in1, in1=0A= cbnz w4, 2f=0A= -1:=0A= - ldxp res0, res1, [x5]=0A= +=0A= + /* RELAXED. */=0A= +1: ldxp res0, res1, [x5]=0A= orn res0, in0, res0=0A= orn res1, in1, res1=0A= stxp w4, res0, res1, [x5]=0A= cbnz w4, 1b=0A= ret=0A= -2:=0A= - ldaxp res0, res1, [x5]=0A= +=0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= +2: ldaxp res0, res1, [x5]=0A= orn res0, in0, res0=0A= orn res1, in1, res1=0A= stlxp w4, res0, res1, [x5]=0A= @@ -413,9 +451,12 @@ END (libat_nand_fetch_16_i1)=0A= ENTRY (libat_test_and_set_16_i1)=0A= mov w2, 1=0A= cbnz w1, 2f=0A= +=0A= + /* RELAXED. */=0A= swpb w0, w2, [x0]=0A= ret=0A= =0A= + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */=0A= 2: swpalb w0, w2, [x0]=0A= ret=0A= END (libat_test_and_set_16_i1)=0A= =0A=