From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2064.outbound.protection.outlook.com [40.107.20.64]) by sourceware.org (Postfix) with ESMTPS id B5C373858410 for ; Fri, 17 Mar 2023 09:56:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B5C373858410 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iAk127+LfF4oTOHgu66arYvteQsXX2QCyOkfk/06KII=; b=bW/1+MDLKFAqgGLavAGJWXzm4aPv6+FIifgZdEqkJuDqunwSPex3+crLCQpKO5KXwTCJ+aXF2OJvci+FV0jwXJ3ylXCRlKHYMpSDUIphnXGFqmaiEIeAgb5M6x7p+X+9lApl2SZun4my2wmpo1pXfOMP90GW8iqchxVYGutf60c= Received: from DU2PR04CA0154.eurprd04.prod.outlook.com (2603:10a6:10:2b0::9) by DU0PR08MB8932.eurprd08.prod.outlook.com (2603:10a6:10:467::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.35; Fri, 17 Mar 2023 09:56:47 +0000 Received: from DBAEUR03FT007.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:2b0:cafe::1a) by DU2PR04CA0154.outlook.office365.com (2603:10a6:10:2b0::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.35 via Frontend Transport; Fri, 17 Mar 2023 09:56:47 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DBAEUR03FT007.mail.protection.outlook.com (100.127.142.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6199.20 via Frontend Transport; Fri, 17 Mar 2023 09:56:47 +0000 Received: ("Tessian outbound c2bcb4c18c29:v135"); Fri, 17 Mar 2023 09:56:47 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 384297e09bab3913 X-CR-MTA-TID: 64aa7808 Received: from a86c2070d45b.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id E55AA69C-46B1-49F0-BEEE-77A1A87C2E4D.1; Fri, 17 Mar 2023 09:56:41 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id a86c2070d45b.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 17 Mar 2023 09:56:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U9BDEwMbuJZ8dyp/BKyBppy12WdJWGHYjktQBMeLyye+YM4eazXgxLPc41P81/fraMpONIYz5a/Cf5HJsKHJE+Y0DNH0BTd9cdP8cfQKoWjaCoGQj3s5Fn0bSqW9dgcIA0uB40gOpMfXDI6qxjOju4QrtGgObzDmbpQ69M2ncsfBXnNuKeBcEXmbWOmc5ugScMx/Rio2+vzRxhRX2tIvwHOYheIeeJmwnYMTxyVYJ6sEAad9/65VjB0F/qeUCo0bzUzO7yTkQwY/rs3xpNg+NRU1/bCF+BUrEURzKm1brgR598y0CaxgWDcOD89iXUbO7paG9x4aACKcnE1hnj1C4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iAk127+LfF4oTOHgu66arYvteQsXX2QCyOkfk/06KII=; b=Le7ZcuAcVAWA3Y1mNZ3sUTkyRiK5ZkO6ZOe2x0wVseN3oJEWpvwh+HVyT+5TYOvyoFcFl9lJwI6JlWtWqVIG5gyFdzA1ViilKDUTY/3o0yIOp7QYjnkqtcmzooPSu1n0JOCuVLM995vrTcFlWORIIDnmQYPu4FUqf196ArOkc7XUVihdsszzN9e4nesHHFpLVX+A/kAGWT4bmD9o5lTZxqmbF9mODHwoXCZOScySQ8ndUTP8uZRTGbgbx+TTmu1ODp3wS5NKCeV+i2SazQrkrSFTMgZqocOCu+JK9ioCva/9vI+wei9QdGyFvR2dvBwhckz3eb3XNt2Ffdi9nITYOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iAk127+LfF4oTOHgu66arYvteQsXX2QCyOkfk/06KII=; b=bW/1+MDLKFAqgGLavAGJWXzm4aPv6+FIifgZdEqkJuDqunwSPex3+crLCQpKO5KXwTCJ+aXF2OJvci+FV0jwXJ3ylXCRlKHYMpSDUIphnXGFqmaiEIeAgb5M6x7p+X+9lApl2SZun4my2wmpo1pXfOMP90GW8iqchxVYGutf60c= Received: from PAXPR08MB6926.eurprd08.prod.outlook.com (2603:10a6:102:138::24) by AS8PR08MB9790.eurprd08.prod.outlook.com (2603:10a6:20b:615::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6178.35; Fri, 17 Mar 2023 09:55:42 +0000 Received: from PAXPR08MB6926.eurprd08.prod.outlook.com ([fe80::2359:6ecb:6536:9acb]) by PAXPR08MB6926.eurprd08.prod.outlook.com ([fe80::2359:6ecb:6536:9acb%7]) with mapi id 15.20.6178.031; Fri, 17 Mar 2023 09:55:41 +0000 From: Kyrylo Tkachov To: Wilco Dijkstra , GCC Patches CC: Richard Sandiford Subject: RE: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891] Thread-Topic: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891] Thread-Index: AQHZR5de3lyELmNywka8opZDiKy6Fa792POCgAEFGnA= Date: Fri, 17 Mar 2023 09:55:41 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: PAXPR08MB6926:EE_|AS8PR08MB9790:EE_|DBAEUR03FT007:EE_|DU0PR08MB8932:EE_ X-MS-Office365-Filtering-Correlation-Id: 27d25015-5c5f-4880-84ef-08db26cdec46 x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: z7RBOMrfOZRm8rnmMUo4kGZSJ2hjbObHhUYsxJHmZhdGVYBBogcubCK/Idw+r6kNnsFxgm5G9kxbmypoHOLEDu2yFuaFYSN+PAsQBspS+z6g75V08hiY/5IWbRV+blT11fHfeZqRm2Bm3XVHz/N13/B2PQ6aVMPeH48zn6d0ZGuhrZ0A4lmGblZSv76hXuQXWPEU4MREHGX/4ufnEHtSWivBILl1EOtTzvvxQoKZ863xpSlEpMwwbnMeGZXsQ81B1/XDHEpjJM+1WQBWBO7YxnXxOsuOTHn4N+lxyZkQRw6Wv1x4Es8PGW7TmS+5BMOj/gXE5GKtw6CH7KyyVbHshT3j6G6sWQs54FiPe4yAiquEbANin25doPiFAYaBQ20l12M4IiQwigPHcEji4gWBV+yPxbUCxyVq/EZ7Gggqf/VLfG1L04+HdJhvFadAhGnSFFmTGEPxA9hpWmkTNGKe1lQ38E4PPa9YIUCT+1sJVJBCVxQrtd9/rzmicRcyVvPZvgkgctWxKZ+4M/eK8nz+rPvN3aehCp3aLDuXlvbGvEw/trj4SMnyyY2Wy29FuB4ljwum6IKVdNcDY2WEJdkIzJ9ecQw5FHBdrrManUreWMOI3B3kyrs6JvELRXOnc+oZZ6z8tSF3cigsFgo9tqHKCGHZRWCEZTfm6RAKRdGHlVqTPbxstMbtfb48Qr0Xo1iEHEAaLkpn7i7xMWTZVHCQcA== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAXPR08MB6926.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(396003)(39860400002)(366004)(346002)(376002)(136003)(451199018)(7696005)(55016003)(71200400001)(9686003)(186003)(6506007)(53546011)(26005)(83380400001)(86362001)(8936002)(5660300002)(52536014)(41300700001)(33656002)(478600001)(2906002)(122000001)(110136005)(38100700002)(64756008)(4326008)(66556008)(76116006)(66476007)(66946007)(66446008)(316002)(8676002)(38070700005);DIR:OUT;SFP:1101; Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB9790 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DBAEUR03FT007.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 856e06dd-bd85-4ee0-f454-08db26cdc4b5 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4YYDs2WjKR1xjVSvhUUuIgTW0kcXD1WdN1ikKsokbBGya9IwsvimML6J5FFWKV6FgfQLlygMxyetf3zxk7uqDNhLJZoQk8Zqb8b+QDItbrrcsycAqR2g1ZW0IsC8FpVWme/UYgAGjh6JATJx0687fypb1PBt3WDyMq97qNJrXpEp5I4yMc1ullQRzctN2fdvHw+xPoBIQC9jvOEBvuKKct0BzHiljl8ZpWf3Dpy0RJ6LDn9d5hy8aYI6A7g9QUAtaMcNmHDKXyvXoOZSPJaR4L/jxOKzTXW1LmU9oPFnwU3Sz/YY4UzZh7aGpGNSzs1JTJMl4DDrXSQV5Zd3Je+S9buR15ZRvBYMsHrRUGxhdQWD+TXsQWYKp8wEVDMh9QSQdcgUH/SY7t0kqUpjcSqb4NtOhL3sBj3/yCxJGmjHJP6GBwcUGjXW65W3s0MtZR8zVrOYtHJHCstaWM/oQi3zFX/H1SfiC0sJebzch+Jg5V3ygmGA2kZlyGbviME9rWkqKuub5Ec4YnflAPQOcsU1xrks0Po08IM7k4xCbW4QKciW3hEtY5omCD1hYP+UtcqHnJUdu/eRGGvZggWWK5P0+ALBL9ko35By564FafXaR0x3OjN9PvekV15PVTBWNXOGjs0DpR18Idkct/AKdpk8mkqaVO4hkNay7+lPSqS6okV8PU3BSKUCrKoqx772v07jeM/3UK6RAqib65AdsW3RAQ== X-Forefront-Antispam-Report: CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230025)(4636009)(396003)(136003)(346002)(39860400002)(376002)(451199018)(46966006)(36840700001)(40470700004)(7696005)(478600001)(186003)(336012)(47076005)(83380400001)(110136005)(70586007)(316002)(70206006)(26005)(9686003)(6506007)(53546011)(36860700001)(41300700001)(8676002)(4326008)(52536014)(8936002)(5660300002)(81166007)(40460700003)(82740400003)(2906002)(356005)(55016003)(82310400005)(86362001)(40480700001)(33656002);DIR:OUT;SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Mar 2023 09:56:47.7096 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 27d25015-5c5f-4880-84ef-08db26cdec46 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DBAEUR03FT007.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8932 X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,FORGED_SPF_HELO,GIT_PATCH_0,KAM_DMARC_NONE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SCC_10_SHORT_WORD_LINES,SCC_5_SHORT_WORD_LINES,SPF_HELO_PASS,SPF_NONE,TXREP,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: > -----Original Message----- > From: Wilco Dijkstra > Sent: Thursday, March 16, 2023 6:22 PM > To: GCC Patches > Cc: Richard Sandiford ; Kyrylo Tkachov > > Subject: Re: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891= ] >=20 > ping >=20 >=20 > From: Wilco Dijkstra > Sent: 23 February 2023 15:11 > To: GCC Patches > Cc: Richard Sandiford ; Kyrylo Tkachov > > Subject: [PATCH] libatomic: Fix SEQ_CST 128-bit atomic load [PR108891] >=20 >=20 > The LSE2 ifunc for 16-byte atomic load requires a barrier before the LDP = - > without it, it effectively has Load-AcquirePC semantics similar to LDAPR, > which is less restrictive than what __ATOMIC_SEQ_CST requires. This patc= h > fixes this and adds comments to make it easier to see which sequence is > used for each case. Use a load/store exclusive loop for store to simplify > testing memory ordering is correct (it is slightly faster too). >=20 > Passes regress, OK for commit? >=20 Ok. Thanks, Kyrill > libatomic/ > PR libgcc/108891 > config/linux/aarch64/atomic_16.S: Fix libat_load_16_i1. > Add comments describing the memory order. >=20 > --- >=20 > diff --git a/libatomic/config/linux/aarch64/atomic_16.S > b/libatomic/config/linux/aarch64/atomic_16.S > index > 732c3534a06678664a252bdbc53652eeab0af506..05439ce394b9653c9bcb582 > 761ff7aaa7c8f9643 100644 > --- a/libatomic/config/linux/aarch64/atomic_16.S > +++ b/libatomic/config/linux/aarch64/atomic_16.S > @@ -72,33 +72,38 @@ name: \ >=20 > ENTRY (libat_load_16_i1) > cbnz w1, 1f > + > + /* RELAXED. */ > ldp res0, res1, [x0] > ret > 1: > - cmp w1, ACQUIRE > - b.hi 2f > + cmp w1, SEQ_CST > + b.eq 2f > + > + /* ACQUIRE/CONSUME (Load-AcquirePC semantics). */ > ldp res0, res1, [x0] > dmb ishld > ret > -2: > + > + /* SEQ_CST. */ > +2: ldar tmp0, [x0] /* Block reordering with Store-Release in= str. */ > ldp res0, res1, [x0] > - dmb ish > + dmb ishld > ret > END (libat_load_16_i1) >=20 >=20 > ENTRY (libat_store_16_i1) > cbnz w4, 1f > + > + /* RELAXED. */ > stp in0, in1, [x0] > ret > -1: > - dmb ish > - stp in0, in1, [x0] > - cmp w4, SEQ_CST > - beq 2f > - ret > -2: > - dmb ish > + > + /* RELEASE/SEQ_CST. */ > +1: ldaxp xzr, tmp0, [x0] > + stlxp w4, in0, in1, [x0] > + cbnz w4, 1b > ret > END (libat_store_16_i1) >=20 > @@ -106,29 +111,33 @@ END (libat_store_16_i1) > ENTRY (libat_exchange_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > stxp w4, in0, in1, [x5] > cbnz w4, 1b > ret > 2: > cmp w4, ACQUIRE > b.hi 4f > -3: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME. */ > +3: ldaxp res0, res1, [x5] > stxp w4, in0, in1, [x5] > cbnz w4, 3b > ret > 4: > cmp w4, RELEASE > b.ne 6f > -5: > - ldxp res0, res1, [x5] > + > + /* RELEASE. */ > +5: ldxp res0, res1, [x5] > stlxp w4, in0, in1, [x5] > cbnz w4, 5b > ret > -6: > - ldaxp res0, res1, [x5] > + > + /* ACQ_REL/SEQ_CST. */ > +6: ldaxp res0, res1, [x5] > stlxp w4, in0, in1, [x5] > cbnz w4, 6b > ret > @@ -142,6 +151,8 @@ ENTRY (libat_compare_exchange_16_i1) > cbz w4, 2f > cmp w4, RELEASE > b.hs 3f > + > + /* ACQUIRE/CONSUME. */ > caspa exp0, exp1, in0, in1, [x0] > 0: > cmp exp0, tmp0 > @@ -153,15 +164,18 @@ ENTRY (libat_compare_exchange_16_i1) > stp exp0, exp1, [x1] > mov x0, 0 > ret > -2: > - casp exp0, exp1, in0, in1, [x0] > + > + /* RELAXED. */ > +2: casp exp0, exp1, in0, in1, [x0] > b 0b > -3: > - b.hi 4f > + > + /* RELEASE. */ > +3: b.hi 4f > caspl exp0, exp1, in0, in1, [x0] > b 0b > -4: > - caspal exp0, exp1, in0, in1, [x0] > + > + /* ACQ_REL/SEQ_CST. */ > +4: caspal exp0, exp1, in0, in1, [x0] > b 0b > END (libat_compare_exchange_16_i1) >=20 > @@ -169,15 +183,17 @@ END (libat_compare_exchange_16_i1) > ENTRY (libat_fetch_add_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > adds tmplo, reslo, inlo > adc tmphi, reshi, inhi > stxp w4, tmp0, tmp1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > adds tmplo, reslo, inlo > adc tmphi, reshi, inhi > stlxp w4, tmp0, tmp1, [x5] > @@ -189,15 +205,17 @@ END (libat_fetch_add_16_i1) > ENTRY (libat_add_fetch_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > adds reslo, reslo, inlo > adc reshi, reshi, inhi > stxp w4, res0, res1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > adds reslo, reslo, inlo > adc reshi, reshi, inhi > stlxp w4, res0, res1, [x5] > @@ -209,15 +227,17 @@ END (libat_add_fetch_16_i1) > ENTRY (libat_fetch_sub_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > subs tmplo, reslo, inlo > sbc tmphi, reshi, inhi > stxp w4, tmp0, tmp1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > subs tmplo, reslo, inlo > sbc tmphi, reshi, inhi > stlxp w4, tmp0, tmp1, [x5] > @@ -229,15 +249,17 @@ END (libat_fetch_sub_16_i1) > ENTRY (libat_sub_fetch_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > subs reslo, reslo, inlo > sbc reshi, reshi, inhi > stxp w4, res0, res1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > subs reslo, reslo, inlo > sbc reshi, reshi, inhi > stlxp w4, res0, res1, [x5] > @@ -249,15 +271,17 @@ END (libat_sub_fetch_16_i1) > ENTRY (libat_fetch_or_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > orr tmp0, res0, in0 > orr tmp1, res1, in1 > stxp w4, tmp0, tmp1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > orr tmp0, res0, in0 > orr tmp1, res1, in1 > stlxp w4, tmp0, tmp1, [x5] > @@ -269,15 +293,17 @@ END (libat_fetch_or_16_i1) > ENTRY (libat_or_fetch_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > orr res0, res0, in0 > orr res1, res1, in1 > stxp w4, res0, res1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > orr res0, res0, in0 > orr res1, res1, in1 > stlxp w4, res0, res1, [x5] > @@ -289,15 +315,17 @@ END (libat_or_fetch_16_i1) > ENTRY (libat_fetch_and_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > and tmp0, res0, in0 > and tmp1, res1, in1 > stxp w4, tmp0, tmp1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > and tmp0, res0, in0 > and tmp1, res1, in1 > stlxp w4, tmp0, tmp1, [x5] > @@ -309,15 +337,17 @@ END (libat_fetch_and_16_i1) > ENTRY (libat_and_fetch_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > and res0, res0, in0 > and res1, res1, in1 > stxp w4, res0, res1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > and res0, res0, in0 > and res1, res1, in1 > stlxp w4, res0, res1, [x5] > @@ -329,15 +359,17 @@ END (libat_and_fetch_16_i1) > ENTRY (libat_fetch_xor_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > eor tmp0, res0, in0 > eor tmp1, res1, in1 > stxp w4, tmp0, tmp1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > eor tmp0, res0, in0 > eor tmp1, res1, in1 > stlxp w4, tmp0, tmp1, [x5] > @@ -349,15 +381,17 @@ END (libat_fetch_xor_16_i1) > ENTRY (libat_xor_fetch_16_i1) > mov x5, x0 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > eor res0, res0, in0 > eor res1, res1, in1 > stxp w4, res0, res1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > eor res0, res0, in0 > eor res1, res1, in1 > stlxp w4, res0, res1, [x5] > @@ -371,15 +405,17 @@ ENTRY (libat_fetch_nand_16_i1) > mvn in0, in0 > mvn in1, in1 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > orn tmp0, in0, res0 > orn tmp1, in1, res1 > stxp w4, tmp0, tmp1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > orn tmp0, in0, res0 > orn tmp1, in1, res1 > stlxp w4, tmp0, tmp1, [x5] > @@ -393,15 +429,17 @@ ENTRY (libat_nand_fetch_16_i1) > mvn in0, in0 > mvn in1, in1 > cbnz w4, 2f > -1: > - ldxp res0, res1, [x5] > + > + /* RELAXED. */ > +1: ldxp res0, res1, [x5] > orn res0, in0, res0 > orn res1, in1, res1 > stxp w4, res0, res1, [x5] > cbnz w4, 1b > ret > -2: > - ldaxp res0, res1, [x5] > + > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > +2: ldaxp res0, res1, [x5] > orn res0, in0, res0 > orn res1, in1, res1 > stlxp w4, res0, res1, [x5] > @@ -413,9 +451,12 @@ END (libat_nand_fetch_16_i1) > ENTRY (libat_test_and_set_16_i1) > mov w2, 1 > cbnz w1, 2f > + > + /* RELAXED. */ > swpb w0, w2, [x0] > ret >=20 > + /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST. */ > 2: swpalb w0, w2, [x0] > ret > END (libat_test_and_set_16_i1)