From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on2083.outbound.protection.outlook.com [40.107.15.83]) by sourceware.org (Postfix) with ESMTPS id E7CCA3858D32 for ; Mon, 16 Oct 2023 12:51:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E7CCA3858D32 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org E7CCA3858D32 Authentication-Results: server2.sourceware.org; arc=fail smtp.remote-ip=40.107.15.83 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1697460663; cv=fail; b=S3rImDgmyzRwvc+eeNR3+35AyydT3LW+skeNg+VHI9CFQRVX068RPhL35Np5m++i69UNFg02rQGTcMpC0nKKwrFQEC8eytgrldzFcl8LBS+vMk4avAsFDpt/MeExKo0rZ4upMSSM93nSOQYFaEvcHy0V/6AkvFwAgqNdZmKNvR4= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1697460663; c=relaxed/simple; bh=6oolYvz8nA3UkVdRrNV6/cJ4VVFt3T1v0hItk6ghy30=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=cUc3yhi4fMh0Ry1J05KqryOsLPn7Lc7EzN8n4UgHIlhFfXWGYKsNF7XQPIlwcnEXTsw5rBb+MNzVtqoFYuLPTh9vKhMXFu4QZBRsha+y7FqBhiPRiiY667PqQTaNbGiAptwYEBRUycM1FSXqiOlmOcSEeaxuRGi3t+NnT1M4sQQ= ARC-Authentication-Results: i=2; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6oolYvz8nA3UkVdRrNV6/cJ4VVFt3T1v0hItk6ghy30=; b=1ONPCaZrXhlgFB57hv96TOnr8ZgelPuXXAS4D9PhUZP9IAAxZYeIt8OkclCc2g6Vrkt9+dFZdGoJbKmpKdm8Ziu/HccBV4+N7obt/EgPFGHxXobc0gptSHSkTLSGmc3e3V3klcDi8BXwlOYljoKKYqeG0OpjdiIQVfxuXuj5xz8= Received: from AS4P191CA0004.EURP191.PROD.OUTLOOK.COM (2603:10a6:20b:5d5::10) by PA4PR08MB6206.eurprd08.prod.outlook.com (2603:10a6:102:ea::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35; Mon, 16 Oct 2023 12:50:58 +0000 Received: from AM4PEPF00027A64.eurprd04.prod.outlook.com (2603:10a6:20b:5d5:cafe::be) by AS4P191CA0004.outlook.office365.com (2603:10a6:20b:5d5::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35 via Frontend Transport; Mon, 16 Oct 2023 12:50:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM4PEPF00027A64.mail.protection.outlook.com (10.167.16.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Mon, 16 Oct 2023 12:50:58 +0000 Received: ("Tessian outbound 0ae75d4034ba:v211"); Mon, 16 Oct 2023 12:50:58 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 1282ce37e967bb39 X-CR-MTA-TID: 64aa7808 Received: from e58ffb0bfc4d.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id F7938D0B-75F5-495E-9609-AA6FDA5E6992.1; Mon, 16 Oct 2023 12:50:51 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id e58ffb0bfc4d.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 16 Oct 2023 12:50:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Hj9wvnyH8LpLvhYLPYDhGB85TqxMJKXSEyNmwIkIflwvT0/S2x07q37YAnGOnZmI+JEt2UYq1T44jtRMe1HqBoOnm1ASyToa3eeXTWbiXMVwkySxB+Z9ZN1aQqAMaF4xac2gKhiFgTLy4SxGL216/FiMIWjErBLjOojoKyOcudNNDh9J5Ezt+1FGIegajNMia7GKW00siWTn3ceSsB8V9Mmky7x9Xxcbp3RQPQ4rRuFTposWWvF1gfkoxEJsevy4aB+SIwZpwigyrPnYNiabB50PrjeHtsUvWfkE/nkhT0bRKN5XpbSX5d1j0a5sL5utAk0IjJTAL/W670++bbogVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6oolYvz8nA3UkVdRrNV6/cJ4VVFt3T1v0hItk6ghy30=; b=dYCLnWpxWQIdLgkmnLQyJ5dmyJ+DrtAVg2ofWTgZrbEnGH8wOjEovDgonoiLUYiTvbO8EJJDf3Cnc5CUn0nqlpdal1WDk9v1Wrt3C9ilOpSfBexDxLiT1HOWZhGBhELfyAp58lEImqnyyqN1tzdO1wG5Qt6jKNUd95YF/uA54UeDDopJfRk8USsRNMBu5Rf1uo1EoHALQF8YSDiNARBYfU1Y8lh90Wmq/MGQejhc5O6Ivt7cCNcc+OcJQXddCXHdU7N8NJazrPDoornDMGKnBe6zSYeXqNhOsFwna/bYml/jcdoRKUUrHMFNhqoGfRuYCqJMBkcdQNEPDt+5c9VA3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6oolYvz8nA3UkVdRrNV6/cJ4VVFt3T1v0hItk6ghy30=; b=1ONPCaZrXhlgFB57hv96TOnr8ZgelPuXXAS4D9PhUZP9IAAxZYeIt8OkclCc2g6Vrkt9+dFZdGoJbKmpKdm8Ziu/HccBV4+N7obt/EgPFGHxXobc0gptSHSkTLSGmc3e3V3klcDi8BXwlOYljoKKYqeG0OpjdiIQVfxuXuj5xz8= Received: from PAWPR08MB8982.eurprd08.prod.outlook.com (2603:10a6:102:33f::20) by VI0PR08MB10827.eurprd08.prod.outlook.com (2603:10a6:800:203::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.34; Mon, 16 Oct 2023 12:50:46 +0000 Received: from PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::31cd:30d1:37a7:3e8]) by PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::31cd:30d1:37a7:3e8%4]) with mapi id 15.20.6886.034; Mon, 16 Oct 2023 12:50:46 +0000 From: Wilco Dijkstra To: GCC Patches CC: Richard Sandiford , Kyrylo Tkachov Subject: Re: [PATCH] libatomic: Enable lock-free 128-bit atomics on AArch64 [PR110061] Thread-Topic: [PATCH] libatomic: Enable lock-free 128-bit atomics on AArch64 [PR110061] Thread-Index: AQHZlXVPCtC3Z5wAcEqXuUSCuN2uu6+NbaXlgB4wgHGALwG51IA+2lx7gDO5jVQ= Date: Mon, 16 Oct 2023 12:50:46 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: PAWPR08MB8982:EE_|VI0PR08MB10827:EE_|AM4PEPF00027A64:EE_|PA4PR08MB6206:EE_ X-MS-Office365-Filtering-Correlation-Id: d3936f81-8635-4cd6-ba57-08dbce468b3b x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 6Nz5Az90J8k5SNpCawNRTiKe0q17gCCQBzaqFGU80RQwFPwkjE+p/NMMGIyETgD6rg35jjaNlfCDFhseuPH25Tw9n/Uv2/efgHnTg0i4JdLAMclzhVmV7whe0oykTw9Vgl6VIw+0UVTAwkVAMWEtTZHxBtqYHALTUXco4QjN/dz9CjCjqZnRm8/a6esJPVy+qFtGoBR4OwKer9FR26CvrmDBl1XqVXcTg3P3L9710kZ1LmU/ueztYLAE+3lBoHRB2qRGoYkg47Ycnh193GfHDrDa5TkdQCISOGAD0W171o1YT6I8ZRaE4/38twW/4kEfFUYteHze88rHqDEKT1HG+z3hMj/Zu4Fz1A8bQyi9jSXdQLxBTVbNGo/H6hVjXf7WSEqXaG5Ft+O+NMGogmzF0kL91gBByoUbaaQKn80cMau0tPOthkjZsLl06TBXy+7w6bJ0VpZbZtn7kgjVFpigiT4E1/PhTx1/bQFH5qSAr5HQ1AEvY37ENq+KaIfG6kmyzl53kIVS8BwDjR1S3Du1Q2CF/S9zdDBE5hqovDl+uusK9LwrCCFZH4LZG0a6DlHAvEaBkk/64P9JyFu82zBGOl1nWoWio1qD0D3yCIyT1kg= X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PAWPR08MB8982.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(376002)(136003)(366004)(346002)(230922051799003)(186009)(1800799009)(451199024)(64100799003)(478600001)(91956017)(71200400001)(64756008)(66446008)(66476007)(66556008)(66946007)(76116006)(54906003)(6916009)(53546011)(26005)(7696005)(41300700001)(6506007)(9686003)(316002)(4326008)(8676002)(8936002)(52536014)(2906002)(5660300002)(33656002)(38070700005)(122000001)(86362001)(83380400001)(38100700002)(55016003);DIR:OUT;SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI0PR08MB10827 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM4PEPF00027A64.eurprd04.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 55426040-7ab8-40f7-131f-08dbce468424 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: V2NqCDK0rtfm3jz621rYw7oCtJEt51zEQwnplG/+4II9w/HkekqWPWSW+YHVcbWOzt4rnVJHwwIshIXTUvh9uzVcUFjqT1uhF8YzNB/TEtw1abDnVNgUWtQVqcZXuvPv1MzDgpz0aPVyKEqOadseZEwuY//VmnDNmedZphyXVxdnAZsxD7+gjvgJkpD4edaZRSIqwC3JONjMHPiMk0h9Tju+rJLM8mMt6CiLRoEgboS814C4hNe4M6dJedCT0EeFue3f9cTsMNzixJx1sYD4Pl4r2neGc8nydVVopREQNvP04PokSvzQzxDAg4FJrztDBRu/Z05TKXnHGARG2glBIPmbjj4RTkOG9dJoGAwEKWJTa5wSjL25Z74G2SOO/LMDPq4Fz6jKJzcvZPGcsmJD1u1gvgeTOqaeL5v8USmOWzQ5AEzwr8uKeByfiz8VSPYUOmgsr4S1cCxsZaGt3VsFknv+BF8dL8bzAKb9jnsmEamPtVmitqZ0IkDlpAJJpdzwSl0gRzdpBx7zYr/LYoZ9xxgXgecM3Sui/zggqFZ3yOs/q4OVIMs8OPnpQLwY3TcFE9NRFLDo3DDybwADtvEqUUEJNdV6gR1ZkxILn2JYmQQkHkdqVWERDHImr243phbOe1UolR8cnt9SRAKviUBr6AFL6xmyc0oQxtjyS+Qf7RBLiZJ/G7ZSMuRSjKu3f0HqCjXC0bbl9vMbsA1pXyL6njHQYOOb7ysALHlSUcTUZXM= X-Forefront-Antispam-Report: CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(346002)(376002)(396003)(136003)(230922051799003)(186009)(1800799009)(82310400011)(64100799003)(451199024)(36840700001)(40470700004)(46966006)(41300700001)(70206006)(478600001)(70586007)(54906003)(6916009)(6506007)(26005)(7696005)(336012)(53546011)(316002)(9686003)(8936002)(4326008)(8676002)(52536014)(2906002)(5660300002)(33656002)(81166007)(86362001)(47076005)(36860700001)(83380400001)(82740400003)(356005)(55016003)(40460700003)(40480700001);DIR:OUT;SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2023 12:50:58.1255 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d3936f81-8635-4cd6-ba57-08dbce468b3b X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM4PEPF00027A64.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR08MB6206 X-Spam-Status: No, score=-9.2 required=5.0 tests=BAYES_00,BODY_8BITS,DKIM_SIGNED,DKIM_VALID,FORGED_SPF_HELO,GIT_PATCH_0,KAM_DMARC_NONE,KAM_SHORT,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,TXREP,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: =A0=0A= =0A= ping=0A= =0A= From: Wilco Dijkstra=0A= Sent: 02 June 2023 18:28=0A= To: GCC Patches =0A= Cc: Richard Sandiford ; Kyrylo Tkachov =0A= Subject: [PATCH] libatomic: Enable lock-free 128-bit atomics on AArch64 [PR= 110061] =0A= =A0=0A= =0A= Enable lock-free 128-bit atomics on AArch64.=A0 This is backwards compatibl= e with=0A= existing binaries, gives better performance than locking atomics and is wha= t=0A= most users expect.=0A= =0A= Note 128-bit atomic loads use a load/store exclusive loop if LSE2 is not su= pported.=0A= This results in an implicit store which is invisible to software as long as= the given=0A= address is writeable (which will be true when using atomics in actual code)= .=0A= =0A= A simple test on an old Cortex-A72 showed 2.7x speedup of 128-bit atomics.= =0A= =0A= Passes regress, OK for commit?=0A= =0A= libatomic/=0A= =A0=A0=A0=A0=A0=A0=A0 PR target/110061=0A= =A0=A0=A0=A0=A0=A0=A0 config/linux/aarch64/atomic_16.S: Implement lock-free= ARMv8.0 atomics.=0A= =A0=A0=A0=A0=A0=A0=A0 config/linux/aarch64/host-config.h: Use atomic_16.S f= or baseline v8.0.=0A= =A0=A0=A0=A0=A0=A0=A0 State we have lock-free atomics.=0A= =0A= ---=0A= =0A= diff --git a/libatomic/config/linux/aarch64/atomic_16.S b/libatomic/config/= linux/aarch64/atomic_16.S=0A= index 05439ce394b9653c9bcb582761ff7aaa7c8f9643..0485c284117edf54f41959d2fab= 9341a9567b1cf 100644=0A= --- a/libatomic/config/linux/aarch64/atomic_16.S=0A= +++ b/libatomic/config/linux/aarch64/atomic_16.S=0A= @@ -22,6 +22,21 @@=0A= =A0=A0=A0 .=A0 */=0A= =A0=0A= =A0=0A= +/* AArch64 128-bit lock-free atomic implementation.=0A= +=0A= +=A0=A0 128-bit atomics are now lock-free for all AArch64 architecture vers= ions.=0A= +=A0=A0 This is backwards compatible with existing binaries and gives bette= r=0A= +=A0=A0 performance than locking atomics.=0A= +=0A= +=A0=A0 128-bit atomic loads use a exclusive loop if LSE2 is not supported.= =0A= +=A0=A0 This results in an implicit store which is invisible to software as= long=0A= +=A0=A0 as the given address is writeable.=A0 Since all other atomics have = explicit=0A= +=A0=A0 writes, this will be true when using atomics in actual code.=0A= +=0A= +=A0=A0 The libat__16 entry points are ARMv8.0.=0A= +=A0=A0 The libat__16_i1 entry points are used when LSE2 is available.= =A0 */=0A= +=0A= +=0A= =A0=A0=A0=A0=A0=A0=A0=A0 .arch=A0=A0 armv8-a+lse=0A= =A0=0A= =A0#define ENTRY(name)=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 \=0A= @@ -37,6 +52,10 @@ name:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 \=0A= =A0=A0=A0=A0=A0=A0=A0=A0 .cfi_endproc;=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 \=0A= =A0=A0=A0=A0=A0=A0=A0=A0 .size name, .-name;=0A= =A0=0A= +#define ALIAS(alias,name)=A0=A0=A0=A0=A0 \=0A= +=A0=A0=A0=A0=A0=A0 .global alias;=A0=A0=A0=A0=A0=A0=A0=A0=A0 \=0A= +=A0=A0=A0=A0=A0=A0 .set alias, name;=0A= +=0A= =A0#define res0 x0=0A= =A0#define res1 x1=0A= =A0#define in0=A0 x2=0A= @@ -70,6 +89,24 @@ name:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 \=0A= =A0#define SEQ_CST 5=0A= =A0=0A= =A0=0A= +ENTRY (libat_load_16)=0A= +=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w1, 2f=0A= +=0A= +=A0=A0=A0=A0=A0=A0 /* RELAXED.=A0 */=0A= +1:=A0=A0=A0=A0 ldxp=A0=A0=A0 res0, res1, [x5]=0A= +=A0=A0=A0=A0=A0=A0 stxp=A0=A0=A0 w4, res0, res1, [x5]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 1b=0A= +=A0=A0=A0=A0=A0=A0 ret=0A= +=0A= +=A0=A0=A0=A0=A0=A0 /* ACQUIRE/CONSUME/SEQ_CST.=A0 */=0A= +2:=A0=A0=A0=A0 ldaxp=A0=A0 res0, res1, [x5]=0A= +=A0=A0=A0=A0=A0=A0 stxp=A0=A0=A0 w4, res0, res1, [x5]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= +=A0=A0=A0=A0=A0=A0 ret=0A= +END (libat_load_16)=0A= +=0A= +=0A= =A0ENTRY (libat_load_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w1, 1f=0A= =A0=0A= @@ -93,6 +130,23 @@ ENTRY (libat_load_16_i1)=0A= =A0END (libat_load_16_i1)=0A= =A0=0A= =A0=0A= +ENTRY (libat_store_16)=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= +=0A= +=A0=A0=A0=A0=A0=A0 /* RELAXED.=A0 */=0A= +1:=A0=A0=A0=A0 ldxp=A0=A0=A0 xzr, tmp0, [x0]=0A= +=A0=A0=A0=A0=A0=A0 stxp=A0=A0=A0 w4, in0, in1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 1b=0A= +=A0=A0=A0=A0=A0=A0 ret=0A= +=0A= +=A0=A0=A0=A0=A0=A0 /* RELEASE/SEQ_CST.=A0 */=0A= +2:=A0=A0=A0=A0 ldxp=A0=A0=A0 xzr, tmp0, [x0]=0A= +=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, in0, in1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= +=A0=A0=A0=A0=A0=A0 ret=0A= +END (libat_store_16)=0A= +=0A= +=0A= =A0ENTRY (libat_store_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 1f=0A= =A0=0A= @@ -101,14 +155,14 @@ ENTRY (libat_store_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= =A0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 /* RELEASE/SEQ_CST.=A0 */=0A= -1:=A0=A0=A0=A0 ldaxp=A0=A0 xzr, tmp0, [x0]=0A= +1:=A0=A0=A0=A0 ldxp=A0=A0=A0 xzr, tmp0, [x0]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, in0, in1, [x0]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 1b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= =A0END (libat_store_16_i1)=0A= =A0=0A= =A0=0A= -ENTRY (libat_exchange_16_i1)=0A= +ENTRY (libat_exchange_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -126,22 +180,55 @@ ENTRY (libat_exchange_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stxp=A0=A0=A0 w4, in0, in1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 3b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -4:=0A= -=A0=A0=A0=A0=A0=A0 cmp=A0=A0=A0=A0 w4, RELEASE=0A= -=A0=A0=A0=A0=A0=A0 b.ne=A0=A0=A0 6f=0A= =A0=0A= -=A0=A0=A0=A0=A0=A0 /* RELEASE.=A0 */=0A= -5:=A0=A0=A0=A0 ldxp=A0=A0=A0 res0, res1, [x5]=0A= +=A0=A0=A0=A0=A0=A0 /* RELEASE/ACQ_REL/SEQ_CST.=A0 */=0A= +4:=A0=A0=A0=A0 ldaxp=A0=A0 res0, res1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, in0, in1, [x5]=0A= -=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 5b=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 4b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= +END (libat_exchange_16)=0A= =A0=0A= -=A0=A0=A0=A0=A0=A0 /* ACQ_REL/SEQ_CST.=A0 */=0A= -6:=A0=A0=A0=A0 ldaxp=A0=A0 res0, res1, [x5]=0A= -=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, in0, in1, [x5]=0A= -=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 6b=0A= +=0A= +ENTRY (libat_compare_exchange_16)=0A= +=A0=A0=A0=A0=A0=A0 ldp=A0=A0=A0=A0 exp0, exp1, [x1]=0A= +=A0=A0=A0=A0=A0=A0 cbz=A0=A0=A0=A0 w4, 3f=0A= +=A0=A0=A0=A0=A0=A0 cmp=A0=A0=A0=A0 w4, RELEASE=0A= +=A0=A0=A0=A0=A0=A0 b.hs=A0=A0=A0 4f=0A= +=0A= +=A0=A0=A0=A0=A0=A0 /* ACQUIRE/CONSUME.=A0 */=0A= +1:=A0=A0=A0=A0 ldaxp=A0=A0 tmp0, tmp1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cmp=A0=A0=A0=A0 tmp0, exp0=0A= +=A0=A0=A0=A0=A0=A0 ccmp=A0=A0=A0 tmp1, exp1, 0, eq=0A= +=A0=A0=A0=A0=A0=A0 bne=A0=A0=A0=A0 2f=0A= +=A0=A0=A0=A0=A0=A0 stxp=A0=A0=A0 w4, in0, in1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 1b=0A= +=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x0, 1=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_exchange_16_i1)=0A= +=0A= +2:=A0=A0=A0=A0 stp=A0=A0=A0=A0 tmp0, tmp1, [x1]=0A= +=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x0, 0=0A= +=A0=A0=A0=A0=A0=A0 ret=0A= +=0A= +=A0=A0=A0=A0=A0=A0 /* RELAXED.=A0 */=0A= +3:=A0=A0=A0=A0 ldxp=A0=A0=A0 tmp0, tmp1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cmp=A0=A0=A0=A0 tmp0, exp0=0A= +=A0=A0=A0=A0=A0=A0 ccmp=A0=A0=A0 tmp1, exp1, 0, eq=0A= +=A0=A0=A0=A0=A0=A0 bne=A0=A0=A0=A0 2b=0A= +=A0=A0=A0=A0=A0=A0 stxp=A0=A0=A0 w4, in0, in1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 3b=0A= +=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x0, 1=0A= +=A0=A0=A0=A0=A0=A0 ret=0A= +=0A= +=A0=A0=A0=A0=A0=A0 /* RELEASE/ACQ_REL/SEQ_CST.=A0 */=0A= +4:=A0=A0=A0=A0 ldaxp=A0=A0 tmp0, tmp1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cmp=A0=A0=A0=A0 tmp0, exp0=0A= +=A0=A0=A0=A0=A0=A0 ccmp=A0=A0=A0 tmp1, exp1, 0, eq=0A= +=A0=A0=A0=A0=A0=A0 bne=A0=A0=A0=A0 2b=0A= +=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, in0, in1, [x0]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 4b=0A= +=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x0, 1=0A= +=A0=A0=A0=A0=A0=A0 ret=0A= +END (libat_compare_exchange_16)=0A= =A0=0A= =A0=0A= =A0ENTRY (libat_compare_exchange_16_i1)=0A= @@ -180,7 +267,7 @@ ENTRY (libat_compare_exchange_16_i1)=0A= =A0END (libat_compare_exchange_16_i1)=0A= =A0=0A= =A0=0A= -ENTRY (libat_fetch_add_16_i1)=0A= +ENTRY (libat_fetch_add_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -199,10 +286,10 @@ ENTRY (libat_fetch_add_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, tmp0, tmp1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_fetch_add_16_i1)=0A= +END (libat_fetch_add_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_add_fetch_16_i1)=0A= +ENTRY (libat_add_fetch_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -221,10 +308,10 @@ ENTRY (libat_add_fetch_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, res0, res1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_add_fetch_16_i1)=0A= +END (libat_add_fetch_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_fetch_sub_16_i1)=0A= +ENTRY (libat_fetch_sub_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -243,10 +330,10 @@ ENTRY (libat_fetch_sub_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, tmp0, tmp1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_fetch_sub_16_i1)=0A= +END (libat_fetch_sub_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_sub_fetch_16_i1)=0A= +ENTRY (libat_sub_fetch_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -265,10 +352,10 @@ ENTRY (libat_sub_fetch_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, res0, res1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_sub_fetch_16_i1)=0A= +END (libat_sub_fetch_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_fetch_or_16_i1)=0A= +ENTRY (libat_fetch_or_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -287,10 +374,10 @@ ENTRY (libat_fetch_or_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, tmp0, tmp1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_fetch_or_16_i1)=0A= +END (libat_fetch_or_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_or_fetch_16_i1)=0A= +ENTRY (libat_or_fetch_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -309,10 +396,10 @@ ENTRY (libat_or_fetch_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, res0, res1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_or_fetch_16_i1)=0A= +END (libat_or_fetch_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_fetch_and_16_i1)=0A= +ENTRY (libat_fetch_and_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -331,10 +418,10 @@ ENTRY (libat_fetch_and_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, tmp0, tmp1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_fetch_and_16_i1)=0A= +END (libat_fetch_and_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_and_fetch_16_i1)=0A= +ENTRY (libat_and_fetch_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -353,10 +440,10 @@ ENTRY (libat_and_fetch_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, res0, res1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_and_fetch_16_i1)=0A= +END (libat_and_fetch_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_fetch_xor_16_i1)=0A= +ENTRY (libat_fetch_xor_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -375,10 +462,10 @@ ENTRY (libat_fetch_xor_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, tmp0, tmp1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_fetch_xor_16_i1)=0A= +END (libat_fetch_xor_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_xor_fetch_16_i1)=0A= +ENTRY (libat_xor_fetch_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2f=0A= =A0=0A= @@ -397,10 +484,10 @@ ENTRY (libat_xor_fetch_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, res0, res1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_xor_fetch_16_i1)=0A= +END (libat_xor_fetch_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_fetch_nand_16_i1)=0A= +ENTRY (libat_fetch_nand_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mvn=A0=A0=A0=A0 in0, in0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mvn=A0=A0=A0=A0 in1, in1=0A= @@ -421,10 +508,10 @@ ENTRY (libat_fetch_nand_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, tmp0, tmp1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_fetch_nand_16_i1)=0A= +END (libat_fetch_nand_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_nand_fetch_16_i1)=0A= +ENTRY (libat_nand_fetch_16)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mvn=A0=A0=A0=A0 in0, in0=0A= =A0=A0=A0=A0=A0=A0=A0=A0 mvn=A0=A0=A0=A0 in1, in1=0A= @@ -445,21 +532,38 @@ ENTRY (libat_nand_fetch_16_i1)=0A= =A0=A0=A0=A0=A0=A0=A0=A0 stlxp=A0=A0 w4, res0, res1, [x5]=0A= =A0=A0=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 2b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_nand_fetch_16_i1)=0A= +END (libat_nand_fetch_16)=0A= =A0=0A= =A0=0A= -ENTRY (libat_test_and_set_16_i1)=0A= -=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 w2, 1=0A= -=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w1, 2f=0A= -=0A= -=A0=A0=A0=A0=A0=A0 /* RELAXED.=A0 */=0A= -=A0=A0=A0=A0=A0=A0 swpb=A0=A0=A0 w0, w2, [x0]=0A= -=A0=A0=A0=A0=A0=A0 ret=0A= +/* __atomic_test_and_set is always inlined, so this entry is unused and=0A= +=A0=A0 only required for completeness.=A0 */=0A= +ENTRY (libat_test_and_set_16)=0A= =A0=0A= -=A0=A0=A0=A0=A0=A0 /* ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST.=A0 */=0A= -2:=A0=A0=A0=A0 swpalb=A0 w0, w2, [x0]=0A= +=A0=A0=A0=A0=A0=A0 /* RELAXED/ACQUIRE/CONSUME/RELEASE/ACQ_REL/SEQ_CST.=A0 = */=0A= +=A0=A0=A0=A0=A0=A0 mov=A0=A0=A0=A0 x5, x0=0A= +1:=A0=A0=A0=A0 ldaxrb=A0 w0, [x5]=0A= +=A0=A0=A0=A0=A0=A0 stlxrb=A0 w4, w2, [x5]=0A= +=A0=A0=A0=A0=A0=A0 cbnz=A0=A0=A0 w4, 1b=0A= =A0=A0=A0=A0=A0=A0=A0=A0 ret=0A= -END (libat_test_and_set_16_i1)=0A= +END (libat_test_and_set_16)=0A= +=0A= +=0A= +/* Alias entry points which are the same in baseline and LSE2.=A0 */=0A= +=0A= +ALIAS (libat_exchange_16_i1, libat_exchange_16)=0A= +ALIAS (libat_fetch_add_16_i1, libat_fetch_add_16)=0A= +ALIAS (libat_add_fetch_16_i1, libat_add_fetch_16)=0A= +ALIAS (libat_fetch_sub_16_i1, libat_fetch_sub_16)=0A= +ALIAS (libat_sub_fetch_16_i1, libat_sub_fetch_16)=0A= +ALIAS (libat_fetch_or_16_i1, libat_fetch_or_16)=0A= +ALIAS (libat_or_fetch_16_i1, libat_or_fetch_16)=0A= +ALIAS (libat_fetch_and_16_i1, libat_fetch_and_16)=0A= +ALIAS (libat_and_fetch_16_i1, libat_and_fetch_16)=0A= +ALIAS (libat_fetch_xor_16_i1, libat_fetch_xor_16)=0A= +ALIAS (libat_xor_fetch_16_i1, libat_xor_fetch_16)=0A= +ALIAS (libat_fetch_nand_16_i1, libat_fetch_nand_16)=0A= +ALIAS (libat_nand_fetch_16_i1, libat_nand_fetch_16)=0A= +ALIAS (libat_test_and_set_16_i1, libat_test_and_set_16)=0A= =A0=0A= =A0=0A= =A0/* GNU_PROPERTY_AARCH64_* macros from elf.h for use in asm code.=A0 */= =0A= diff --git a/libatomic/config/linux/aarch64/host-config.h b/libatomic/confi= g/linux/aarch64/host-config.h=0A= index bea26825b4f75bb8ff348ab4b5fc45f4a5bd561e..851c78c01cd643318aaa52929ce= 4550266238b79 100644=0A= --- a/libatomic/config/linux/aarch64/host-config.h=0A= +++ b/libatomic/config/linux/aarch64/host-config.h=0A= @@ -35,10 +35,19 @@=0A= =A0#endif=0A= =A0#define IFUNC_NCOND(N)=A0 (1)=0A= =A0=0A= -#if N =3D=3D 16 && IFUNC_ALT !=3D 0=0A= +#endif /* HAVE_IFUNC */=0A= +=0A= +/* All 128-bit atomic functions are defined in aarch64/atomic_16.S.=A0 */= =0A= +#if N =3D=3D 16=0A= =A0# define DONE 1=0A= =A0#endif=0A= =A0=0A= -#endif /* HAVE_IFUNC */=0A= +/* State we have lock-free 128-bit atomics.=A0 */=0A= +#undef FAST_ATOMIC_LDST_16=0A= +#define FAST_ATOMIC_LDST_16=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 1=0A= +#undef MAYBE_HAVE_ATOMIC_CAS_16=0A= +#define MAYBE_HAVE_ATOMIC_CAS_16=A0=A0=A0=A0=A0=A0 1=0A= +#undef MAYBE_HAVE_ATOMIC_EXCHANGE_16=0A= +#define MAYBE_HAVE_ATOMIC_EXCHANGE_16=A0 1=0A= =A0=0A= =A0#include_next =