From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2059.outbound.protection.outlook.com [40.107.22.59]) by sourceware.org (Postfix) with ESMTPS id 0EC6C385841A for ; Wed, 24 Jan 2024 17:19:16 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0EC6C385841A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 0EC6C385841A Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=40.107.22.59 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1706116766; cv=pass; b=cYkCmpqaIsSEg2befsK+lDHYgbAogpa6L2ydxxxyv19YX9yLg/Z9abKmI4wIfFC/ho6pEh8gB2+NaBq79twrz7dVVbw9jPD8IOmw4cy8z9tpcmIhxRHMSuqlwNjhdAmr9cSi8jarESVyhhTxZQ1KtEDYX8CUvmMUbwCNrRK+xA8= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1706116766; c=relaxed/simple; bh=5Y5zB2qBhmlgXKdHZrMhzCazaoyvriuOv9NHfskMpaA=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=KnS0+kIMviYs5XTzqatWssbeQiZ+YPKUaee1LLJoc3jFFMLMstXQHQhlNRZDHsixExjXHLq8wAHavRulAz9hyMzE9cCWRv/jW8hrj3BVgndW3WYctqa1MIL9lOM/lYBuL08RIZib3xNbPfPb8Ocrc+T8X5gxs0W6oH6iWhP6Ew0= ARC-Authentication-Results: i=3; server2.sourceware.org ARC-Seal: i=2; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=pass; b=nu9hNrgRIq7cTsEq41yXxJTwIAUVGeqUW5HE3lUSnDHm+7DRuQd8uAaastLXbhy0YcKQOMh3Kbh92EBFWfc/L3rrTQoPKh3fSz1hEo9MXcBnYj5ukBVwIBIz0MtyG8JvsSSA+4tFwipcVtF1AUDNUISAyh7d+Vb16xw8INaaK2LcSQrRAFVzcWIl9BDpM8fs97iv8/BnXDf98i8eAQQIk3F1vw4LdcZ/fjOyl8Pyv9ymJsGrkU3iz9vDuthOspvGgO7SBTCag7U1TuR9y8qS40/mo2+HeHAq+/PhPOFecHmbB6yRR0wcztm6uTSkvcincv1DAJNENB7EuN2MDQv91A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AZHAzCGmdWnpw1ykdRnOHLKaWz4X5cAyy5KqcFn6XFk=; b=XrgAXE/L1Kmn4M1eEQQB6KfgedyCaiexk1JzaGabzec89iBoyJ/fndlECMKOOsiTEjRwoCXeJnAG3ACNgC6xPuuFkKXEMwN92jUtar4jTS1MfA+e/qZfD0+7GCi2aMfY++hxfsqDsby3m8lR5l3YcjzZ0ZEvP7wXRbMefJpFGP5CbP4ebhSL+AQL4zqlDkMDMOhBb3ZSz+2CaWgR8foZN7Z2bDL8C5jXBXckmQD12GSnbYm7AiH1JSM4mZEibqmPJ+YbiWnoQ/pF824ZUYXtyHMvMnZ2AHX1Gsl47sZT/XYDliNUOkarxb6Exu8HXkNNZFCuwDgsBiup4WWG8yuDwQ== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 63.35.35.123) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AZHAzCGmdWnpw1ykdRnOHLKaWz4X5cAyy5KqcFn6XFk=; b=iCFNDy7SKuECpCSKhtOtSNkYPjPa2Qk5uIaTrYmIKL/nEn4eSbQ22MkHS1mR/Lox4SVlvw2vun8G97q1R7POBP71nz1SudAgNGr0KmmH026LyPCmcyGs6Tp9D00+MiD8d7VCLZUBOjT5Y2l9QgmDhQIl1uem39LuNGkYdEym1BA= Received: from AM5PR04CA0026.eurprd04.prod.outlook.com (2603:10a6:206:1::39) by PAWPR08MB10212.eurprd08.prod.outlook.com (2603:10a6:102:369::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22; Wed, 24 Jan 2024 17:19:08 +0000 Received: from AM2PEPF0001C710.eurprd05.prod.outlook.com (2603:10a6:206:1:cafe::c5) by AM5PR04CA0026.outlook.office365.com (2603:10a6:206:1::39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22 via Frontend Transport; Wed, 24 Jan 2024 17:19:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM2PEPF0001C710.mail.protection.outlook.com (10.167.16.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7202.16 via Frontend Transport; Wed, 24 Jan 2024 17:19:08 +0000 Received: ("Tessian outbound c4f080b252bb:v228"); Wed, 24 Jan 2024 17:19:08 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: a0bf02e583b0e9b8 X-CR-MTA-TID: 64aa7808 Received: from c8f5cfffaf7e.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 3348E4BB-D08B-4984-B725-85EB0047193B.1; Wed, 24 Jan 2024 17:19:01 +0000 Received: from EUR03-AM7-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id c8f5cfffaf7e.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 24 Jan 2024 17:19:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YWXxnB8nlarBfWGDouxF54HAAMHw4P0PCBNs4H8lmu+eOsHRZigorM2It7RP2yx4ZrR2CiufRiEyvO+a02CTDKhQu28h1g0AjqwQxfm392IzCwboPEMSi9jQU344vrolIy7WVSP+yylwddfd7Op46PZ1HPeji5kUBBWHdM0ouKDBUV6sQqUUfiLXSupkbFcTExUrR3bzIyQxnjoeHmdyMKNatd9mzp8rqYMfw5ZkGfoKEnrhC9xDohSM8jvdfubNHgNxId0EnRWNqGSKk+g2mC37y+tAm/CUtawylm/oDP43KeTAA4CN0BUpoxVOhvPpOKEmyLSYhsA9xZFCRuCnTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AZHAzCGmdWnpw1ykdRnOHLKaWz4X5cAyy5KqcFn6XFk=; b=d2/UePfcgKPWrpFvImlceSqNypgfw4NrHN0t+XXm7igmbv+u4Ya8j7ollSQ6UJIPis6k1XWsThcjgsXQX3QB9MNGZJ6Jvuu3J97H0dkinwHuaDG8SxdBgT8HI5J8eGZyixsUnbKKM5M9CJukCQzvifE0GCakdmSvnQabh9sbiWGss4Hw6d6r6DdscQvi8bwuh3dCZ0a/IeD7yMwKh3cH5Gq5ig+lo4xBCANYLR9bk6FfZJsg1E1mE7kdnMhQ0XMKHe+NkBil3mQXfaYXc8IrpUDHgGTH43f8J7ISQZ/fyIZ1N5NUj3sXugBjT0RRk2JLo550AQ+DECTA6nfwT7r4fA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 40.67.248.234) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AZHAzCGmdWnpw1ykdRnOHLKaWz4X5cAyy5KqcFn6XFk=; b=iCFNDy7SKuECpCSKhtOtSNkYPjPa2Qk5uIaTrYmIKL/nEn4eSbQ22MkHS1mR/Lox4SVlvw2vun8G97q1R7POBP71nz1SudAgNGr0KmmH026LyPCmcyGs6Tp9D00+MiD8d7VCLZUBOjT5Y2l9QgmDhQIl1uem39LuNGkYdEym1BA= Received: from AM0PR04CA0069.eurprd04.prod.outlook.com (2603:10a6:208:1::46) by DU0PR08MB7542.eurprd08.prod.outlook.com (2603:10a6:10:321::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7202.31; Wed, 24 Jan 2024 17:18:59 +0000 Received: from AM3PEPF00009B9B.eurprd04.prod.outlook.com (2603:10a6:208:1:cafe::df) by AM0PR04CA0069.outlook.office365.com (2603:10a6:208:1::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22 via Frontend Transport; Wed, 24 Jan 2024 17:18:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 40.67.248.234) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 40.67.248.234 as permitted sender) receiver=protection.outlook.com; client-ip=40.67.248.234; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (40.67.248.234) by AM3PEPF00009B9B.mail.protection.outlook.com (10.167.16.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7202.16 via Frontend Transport; Wed, 24 Jan 2024 17:18:59 +0000 Received: from AZ-NEU-EX04.Arm.com (10.251.24.32) by AZ-NEU-EX04.Arm.com (10.251.24.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Wed, 24 Jan 2024 17:18:57 +0000 Received: from e125768.cambridge.arm.com (10.2.78.50) by mail.arm.com (10.251.24.32) with Microsoft SMTP Server id 15.1.2507.32 via Frontend Transport; Wed, 24 Jan 2024 17:18:57 +0000 From: Victor Do Nascimento To: CC: , , , Victor Do Nascimento Subject: [PATCH v4 3/4] libatomic: Enable LSE128 128-bit atomics for armv9.4-a Date: Wed, 24 Jan 2024 17:17:32 +0000 Message-ID: <20240124171853.3112540-4-victor.donascimento@arm.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240124171853.3112540-1-victor.donascimento@arm.com> References: <20240124171853.3112540-1-victor.donascimento@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: AM3PEPF00009B9B:EE_|DU0PR08MB7542:EE_|AM2PEPF0001C710:EE_|PAWPR08MB10212:EE_ X-MS-Office365-Filtering-Correlation-Id: 39f1dc2c-5239-429d-694c-08dc1d009341 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 1k8zTZy5/UJ8vvv2b7FE6yQrsBK4XK6s3xn5RnTbAqw3mSluf6oOv28yCji9ugxqRXsj7gt+djX7hsRpgdu16il0bzAfAEv9CmIu/lO/sakHAMQ/hLoEpYNFCBo4cP69uQtjC3taHXFA2VanJI1yM47LbALF/j+P4x44cFWxrW8A4H/4Cy8mXx6UelVwM+ue5U+2M/osTsLHPjkp+zQt4Asrgwnk2/MsX8WYEQ0u2k+Kb8IBrf72Z3bbpvUYeCIa66Whoh6E4Pi/vkzIY86VYcqycq3oJ3R545weH2c0c3lqqfIsCbtd3akv6Dw5J9aRgyutE8SvLW60YrQFyds8DjWCZSfhuAIswuMCYilg6MTeKOorLs+hk2I6F/7V33DEKvRWtag6/lSFX8szs8lyh4rrE0DAGiBIBR9wPBuq1mizyIwOOL7RJjvfIyojmHnSqpSwaRtVTMzgALAU9OY1OOUGlzyryRBB2YWYX90le8LXLt/KoPF9Nhzf3TW353474l780mGIeg+cup7/dzaXrEi56SpQEn7cVBiQV5zZY9iXXShzN7Cuuq/bz0g66/WInrv7mOBRsz13v1rfm3NsT9UhSdwHXX8hjYs3PPDNSQOE5okwuMzHRFukwR7gbtJjc80bWKkeQxrZ7N5GcEgbFXfnBfhKqqkq+/t+X0egxiU75wj8Vh70vV5XFLwcyCw18km0k0Gzx242sSLkOEkv8Mpxv+u0qYg71JGhLwT4TTx/uPmPkIHlW5Lw1Ar99/CqGauHlBqC+E0acIXmyM89lA== X-Forefront-Antispam-Report-Untrusted: CIP:40.67.248.234;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:nebula.arm.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(396003)(346002)(39860400002)(376002)(136003)(230922051799003)(82310400011)(186009)(1800799012)(64100799003)(451199024)(36840700001)(40470700004)(46966006)(70206006)(4326008)(6916009)(356005)(54906003)(70586007)(81166007)(26005)(316002)(40460700003)(8936002)(40480700001)(8676002)(47076005)(83380400001)(7696005)(6666004)(2616005)(478600001)(5660300002)(36756003)(30864003)(41300700001)(2906002)(36860700001)(336012)(426003)(1076003)(86362001)(82740400003)(36900700001);DIR:OUT;SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB7542 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM2PEPF0001C710.eurprd05.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: e6914d27-5abe-4170-dd47-08dc1d008d8d X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: r3Hs2O3LEPjVvXhc9seY4AY+p7/ydvsYH49kPUlbAMQfs64FGCTunriFIsbz64vRP/p9xqDU6U6ibn8aOoJGn6UxLD1Sy6qkEkT/FP13IN0lWUaT0EHi+4frjKxk/LfNWxbmn22duCh/rZPkkeDzSt2mKGz31bziR1WHXYyFaE6rkmja8id6c29J0O91Ha7rim+mXYs70XgGjLf9R9rlKnI8Igoj4KgaVXBWqOsvUUMTCa0RXaXbzQ4AC9+txxjc22jeFWInA0LEfDnnXRc/ln49MlsbMCRgiZwGieihDqFcN73oAsyF8W0/hSKhjkzhR6VH5S2Lrfq7sEPJpU+ZvJHnH31OCSY5UO8wlsJuBToJsa3AC+oBRr+yIcECtZBrQNrCk63GRumrPSXZMapS+52yWywO4DY+y2Vv5IWq4wE23yGzNWpP8SOs9KKuYRtMGLv/qxHMD8Q2HcBA6MlwzwGAhc+B7TUAIMAIZ+EOHTHBerPBGdBqGD+qqrYVy47SWumXdFgOqVdpUcdmWJf4hXO7BnWiYheuJp8Bz9ADMKTto9Q64bS59AaBJRmee28jQoksuCfjrjyflbz0LVeqZxL+ei0by6UDJBwmmY/Si1t8dY07WZmgNE/eCgau+kQr1Zs2WPcc21f1Px7VAstwd64bxjnJlc1vF/Mm15mKSgtgHJvK6hM40t11qPjtto06EWPXEpxAjcQn5EGTGyUcY5WHrugjTP+akJw9PfFhRtF2jCsA//CZJYkiSbiPINsx X-Forefront-Antispam-Report: CIP:63.35.35.123;CTRY:IE;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:64aa7808-outbound-1.mta.getcheckrecipient.com;PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(396003)(39860400002)(376002)(346002)(230922051799003)(186009)(451199024)(1800799012)(82310400011)(64100799003)(36840700001)(40470700004)(46966006)(7696005)(40460700003)(40480700001)(1076003)(2616005)(82740400003)(6666004)(26005)(47076005)(426003)(83380400001)(336012)(36860700001)(478600001)(8936002)(81166007)(70206006)(316002)(70586007)(6916009)(54906003)(8676002)(4326008)(5660300002)(86362001)(41300700001)(2906002)(36756003)(30864003);DIR:OUT;SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2024 17:19:08.6729 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 39f1dc2c-5239-429d-694c-08dc1d009341 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[63.35.35.123];Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM2PEPF0001C710.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10212 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,FORGED_SPF_HELO,GIT_PATCH_0,KAM_DMARC_NONE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SCC_5_SHORT_WORD_LINES,SPF_HELO_PASS,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: The armv9.4-a architectural revision adds three new atomic operations associated with the LSE128 feature: * LDCLRP - Atomic AND NOT (bitclear) of a location with 128-bit value held in a pair of registers, with original data loaded into the same 2 registers. * LDSETP - Atomic OR (bitset) of a location with 128-bit value held in a pair of registers, with original data loaded into the same 2 registers. * SWPP - Atomic swap of one 128-bit value with 128-bit value held in a pair of registers. It is worth noting that in keeping with existing 128-bit atomic operations in `atomic_16.S', we have chosen to merge certain less-restrictive orderings into more restrictive ones. This is done to minimize the number of branches in the atomic functions, minimizing both the likelihood of branch mispredictions and, in keeping code small, limit the need for extra fetch cycles. Past benchmarking has revealed that acquire is typically slightly faster than release (5-10%), such that for the most frequently used atomics (CAS and SWP) it makes sense to add support for acquire, as well as release. Likewise, it was identified that combining acquire and release typically results in little to no penalty, such that it is of negligible benefit to distinguish between release and acquire-release, making the combining release/acq_rel/seq_cst a worthwhile design choice. This patch adds the logic required to make use of these when the architectural feature is present and a suitable assembler available. In order to do this, the following changes are made: 1. Add a configure-time check to check for LSE128 support in the assembler. 2. Edit host-config.h so that when N == 16, nifunc = 2. 3. Where available due to LSE128, implement the second ifunc, making use of the novel instructions. 4. For atomic functions unable to make use of these new instructions, define a new alias which causes the _i1 function variant to point ahead to the corresponding _i2 implementation. libatomic/ChangeLog: * Makefile.am (AM_CPPFLAGS): add conditional setting of -DHAVE_FEAT_LSE128. * acinclude.m4 (LIBAT_TEST_FEAT_AARCH64_LSE128): New. * config/linux/aarch64/atomic_16.S (LSE128): New macro definition. (libat_exchange_16): New LSE128 variant. (libat_fetch_or_16): Likewise. (libat_or_fetch_16): Likewise. (libat_fetch_and_16): Likewise. (libat_and_fetch_16): Likewise. * config/linux/aarch64/host-config.h (IFUNC_COND_2): New. (IFUNC_NCOND): Add operand size checking. (has_lse2): Renamed from `ifunc1`. (has_lse128): New. (HWCAP2_LSE128): Likewise. * libatomic/configure.ac: Add call to LIBAT_TEST_FEAT_AARCH64_LSE128. * configure (ac_subst_vars): Regenerated via autoreconf. * libatomic/Makefile.in: Likewise. * libatomic/auto-config.h.in: Likewise. --- libatomic/Makefile.am | 3 + libatomic/Makefile.in | 1 + libatomic/acinclude.m4 | 19 +++ libatomic/auto-config.h.in | 3 + libatomic/config/linux/aarch64/atomic_16.S | 170 ++++++++++++++++++- libatomic/config/linux/aarch64/host-config.h | 42 ++++- libatomic/configure | 61 ++++++- libatomic/configure.ac | 3 + 8 files changed, 293 insertions(+), 9 deletions(-) diff --git a/libatomic/Makefile.am b/libatomic/Makefile.am index cfad90124f9..0623a0bf2d1 100644 --- a/libatomic/Makefile.am +++ b/libatomic/Makefile.am @@ -130,6 +130,9 @@ libatomic_la_LIBADD = $(foreach s,$(SIZES),$(addsuffix _$(s)_.lo,$(SIZEOBJS))) ## On a target-specific basis, include alternates to be selected by IFUNC. if HAVE_IFUNC if ARCH_AARCH64_LINUX +if ARCH_AARCH64_HAVE_LSE128 +AM_CPPFLAGS = -DHAVE_FEAT_LSE128 +endif IFUNC_OPTIONS = -march=armv8-a+lse libatomic_la_LIBADD += $(foreach s,$(SIZES),$(addsuffix _$(s)_1_.lo,$(SIZEOBJS))) libatomic_la_SOURCES += atomic_16.S diff --git a/libatomic/Makefile.in b/libatomic/Makefile.in index dc2330b91fd..cd48fa21334 100644 --- a/libatomic/Makefile.in +++ b/libatomic/Makefile.in @@ -452,6 +452,7 @@ M_SRC = $(firstword $(filter %/$(M_FILE), $(all_c_files))) libatomic_la_LIBADD = $(foreach s,$(SIZES),$(addsuffix \ _$(s)_.lo,$(SIZEOBJS))) $(am__append_1) $(am__append_3) \ $(am__append_4) $(am__append_5) +@ARCH_AARCH64_HAVE_LSE128_TRUE@@ARCH_AARCH64_LINUX_TRUE@@HAVE_IFUNC_TRUE@AM_CPPFLAGS = -DHAVE_FEAT_LSE128 @ARCH_AARCH64_LINUX_TRUE@@HAVE_IFUNC_TRUE@IFUNC_OPTIONS = -march=armv8-a+lse @ARCH_ARM_LINUX_TRUE@@HAVE_IFUNC_TRUE@IFUNC_OPTIONS = -march=armv7-a+fp -DHAVE_KERNEL64 @ARCH_I386_TRUE@@HAVE_IFUNC_TRUE@IFUNC_OPTIONS = -march=i586 diff --git a/libatomic/acinclude.m4 b/libatomic/acinclude.m4 index f35ab5b60a5..d4f13174e2c 100644 --- a/libatomic/acinclude.m4 +++ b/libatomic/acinclude.m4 @@ -83,6 +83,25 @@ AC_DEFUN([LIBAT_TEST_ATOMIC_BUILTIN],[ ]) ]) +dnl +dnl Test if the host assembler supports armv9.4-a LSE128 isns. +dnl +AC_DEFUN([LIBAT_TEST_FEAT_AARCH64_LSE128],[ + AC_CACHE_CHECK([for armv9.4-a LSE128 insn support], + [libat_cv_have_feat_lse128],[ + AC_LANG_CONFTEST([AC_LANG_PROGRAM([],[asm(".arch armv9-a+lse128")])]) + if AC_TRY_EVAL(ac_compile); then + eval libat_cv_have_feat_lse128=yes + else + eval libat_cv_have_feat_lse128=no + fi + rm -f conftest* + ]) + LIBAT_DEFINE_YESNO([HAVE_FEAT_LSE128], [$libat_cv_have_feat_lse128], + [Have LSE128 support for 16 byte integers.]) + AM_CONDITIONAL([ARCH_AARCH64_HAVE_LSE128], [test x$libat_cv_have_feat_lse128 = xyes]) +]) + dnl dnl Test if we have __atomic_load and __atomic_store for mode $1, size $2 dnl diff --git a/libatomic/auto-config.h.in b/libatomic/auto-config.h.in index ab3424a759e..7c78933b07d 100644 --- a/libatomic/auto-config.h.in +++ b/libatomic/auto-config.h.in @@ -105,6 +105,9 @@ /* Define to 1 if you have the header file. */ #undef HAVE_DLFCN_H +/* Have LSE128 support for 16 byte integers. */ +#undef HAVE_FEAT_LSE128 + /* Define to 1 if you have the header file. */ #undef HAVE_FENV_H diff --git a/libatomic/config/linux/aarch64/atomic_16.S b/libatomic/config/linux/aarch64/atomic_16.S index 16a42925903..979ed8498cd 100644 --- a/libatomic/config/linux/aarch64/atomic_16.S +++ b/libatomic/config/linux/aarch64/atomic_16.S @@ -35,12 +35,17 @@ writes, this will be true when using atomics in actual code. The libat__16 entry points are ARMv8.0. - The libat__16_i1 entry points are used when LSE2 is available. */ - + The libat__16_i1 entry points are used when LSE128 is available. + The libat__16_i2 entry points are used when LSE2 is available. */ +#if HAVE_FEAT_LSE128 + .arch armv9-a+lse128 +#else .arch armv8-a+lse +#endif -#define LSE2(NAME) NAME##_i1 +#define LSE128(NAME) NAME##_i1 +#define LSE2(NAME) NAME##_i2 #define CORE(NAME) NAME #define ENTRY(NAME) ENTRY_FEAT1 (NAME) @@ -206,6 +211,31 @@ ENTRY (libat_exchange_16) END (libat_exchange_16) +#if HAVE_FEAT_LSE128 +ENTRY_FEAT (libat_exchange_16, LSE128) + mov tmp0, x0 + mov res0, in0 + mov res1, in1 + cbnz w4, 1f + + /* RELAXED. */ + swpp res0, res1, [tmp0] + ret +1: + cmp w4, ACQUIRE + b.hi 2f + + /* ACQUIRE/CONSUME. */ + swppa res0, res1, [tmp0] + ret + + /* RELEASE/ACQ_REL/SEQ_CST. */ +2: swppal res0, res1, [tmp0] + ret +END_FEAT (libat_exchange_16, LSE128) +#endif + + ENTRY (libat_compare_exchange_16) ldp exp0, exp1, [x1] cbz w4, 3f @@ -399,6 +429,31 @@ ENTRY (libat_fetch_or_16) END (libat_fetch_or_16) +#if HAVE_FEAT_LSE128 +ENTRY_FEAT (libat_fetch_or_16, LSE128) + mov tmp0, x0 + mov res0, in0 + mov res1, in1 + cbnz w4, 1f + + /* RELAXED. */ + ldsetp res0, res1, [tmp0] + ret +1: + cmp w4, ACQUIRE + b.hi 2f + + /* ACQUIRE/CONSUME. */ + ldsetpa res0, res1, [tmp0] + ret + + /* RELEASE/ACQ_REL/SEQ_CST. */ +2: ldsetpal res0, res1, [tmp0] + ret +END_FEAT (libat_fetch_or_16, LSE128) +#endif + + ENTRY (libat_or_fetch_16) mov x5, x0 cbnz w4, 2f @@ -421,6 +476,36 @@ ENTRY (libat_or_fetch_16) END (libat_or_fetch_16) +#if HAVE_FEAT_LSE128 +ENTRY_FEAT (libat_or_fetch_16, LSE128) + cbnz w4, 1f + mov tmp0, in0 + mov tmp1, in1 + + /* RELAXED. */ + ldsetp in0, in1, [x0] + orr res0, in0, tmp0 + orr res1, in1, tmp1 + ret +1: + cmp w4, ACQUIRE + b.hi 2f + + /* ACQUIRE/CONSUME. */ + ldsetpa in0, in1, [x0] + orr res0, in0, tmp0 + orr res1, in1, tmp1 + ret + + /* RELEASE/ACQ_REL/SEQ_CST. */ +2: ldsetpal in0, in1, [x0] + orr res0, in0, tmp0 + orr res1, in1, tmp1 + ret +END_FEAT (libat_or_fetch_16, LSE128) +#endif + + ENTRY (libat_fetch_and_16) mov x5, x0 cbnz w4, 2f @@ -443,6 +528,32 @@ ENTRY (libat_fetch_and_16) END (libat_fetch_and_16) +#if HAVE_FEAT_LSE128 +ENTRY_FEAT (libat_fetch_and_16, LSE128) + mov tmp0, x0 + mvn res0, in0 + mvn res1, in1 + cbnz w4, 1f + + /* RELAXED. */ + ldclrp res0, res1, [tmp0] + ret + +1: + cmp w4, ACQUIRE + b.hi 2f + + /* ACQUIRE/CONSUME. */ + ldclrpa res0, res1, [tmp0] + ret + + /* RELEASE/ACQ_REL/SEQ_CST. */ +2: ldclrpal res0, res1, [tmp0] + ret +END_FEAT (libat_fetch_and_16, LSE128) +#endif + + ENTRY (libat_and_fetch_16) mov x5, x0 cbnz w4, 2f @@ -465,6 +576,37 @@ ENTRY (libat_and_fetch_16) END (libat_and_fetch_16) +#if HAVE_FEAT_LSE128 +ENTRY_FEAT (libat_and_fetch_16, LSE128) + mvn tmp0, in0 + mvn tmp0, in1 + cbnz w4, 1f + + /* RELAXED. */ + ldclrp tmp0, tmp1, [x0] + and res0, tmp0, in0 + and res1, tmp1, in1 + ret + +1: + cmp w4, ACQUIRE + b.hi 2f + + /* ACQUIRE/CONSUME. */ + ldclrpa tmp0, tmp1, [x0] + and res0, tmp0, in0 + and res1, tmp1, in1 + ret + + /* RELEASE/ACQ_REL/SEQ_CST. */ +2: ldclrpal tmp0, tmp1, [x5] + and res0, tmp0, in0 + and res1, tmp1, in1 + ret +END_FEAT (libat_and_fetch_16, LSE128) +#endif + + ENTRY (libat_fetch_xor_16) mov x5, x0 cbnz w4, 2f @@ -570,6 +712,28 @@ ENTRY (libat_test_and_set_16) END (libat_test_and_set_16) +/* Alias entry points which are the same in LSE2 and LSE128. */ + +#if !HAVE_FEAT_LSE128 +ALIAS (libat_exchange_16, LSE128, LSE2) +ALIAS (libat_fetch_or_16, LSE128, LSE2) +ALIAS (libat_fetch_and_16, LSE128, LSE2) +ALIAS (libat_or_fetch_16, LSE128, LSE2) +ALIAS (libat_and_fetch_16, LSE128, LSE2) +#endif +ALIAS (libat_load_16, LSE128, LSE2) +ALIAS (libat_store_16, LSE128, LSE2) +ALIAS (libat_compare_exchange_16, LSE128, LSE2) +ALIAS (libat_fetch_add_16, LSE128, LSE2) +ALIAS (libat_add_fetch_16, LSE128, LSE2) +ALIAS (libat_fetch_sub_16, LSE128, LSE2) +ALIAS (libat_sub_fetch_16, LSE128, LSE2) +ALIAS (libat_fetch_xor_16, LSE128, LSE2) +ALIAS (libat_xor_fetch_16, LSE128, LSE2) +ALIAS (libat_fetch_nand_16, LSE128, LSE2) +ALIAS (libat_nand_fetch_16, LSE128, LSE2) +ALIAS (libat_test_and_set_16, LSE128, LSE2) + /* Alias entry points which are the same in baseline and LSE2. */ ALIAS (libat_exchange_16, LSE2, CORE) diff --git a/libatomic/config/linux/aarch64/host-config.h b/libatomic/config/linux/aarch64/host-config.h index 8fd4fe3321a..1bc7d839232 100644 --- a/libatomic/config/linux/aarch64/host-config.h +++ b/libatomic/config/linux/aarch64/host-config.h @@ -37,14 +37,17 @@ typedef struct __ifunc_arg_t { #ifdef HWCAP_USCAT # if N == 16 -# define IFUNC_COND_1 ifunc1 (hwcap, features) +# define IFUNC_COND_1 (has_lse128 (hwcap, features)) +# define IFUNC_COND_2 (has_lse2 (hwcap, features)) +# define IFUNC_NCOND(N) 2 # else -# define IFUNC_COND_1 (hwcap & HWCAP_ATOMICS) +# define IFUNC_COND_1 (hwcap & HWCAP_ATOMICS) +# define IFUNC_NCOND(N) 1 # endif #else # define IFUNC_COND_1 (false) +# define IFUNC_NCOND(N) 1 #endif -#define IFUNC_NCOND(N) (1) #endif /* HAVE_IFUNC */ @@ -59,7 +62,7 @@ typedef struct __ifunc_arg_t { #define MIDR_PARTNUM(midr) (((midr) >> 4) & 0xfff) static inline bool -ifunc1 (unsigned long hwcap, const __ifunc_arg_t *features) +has_lse2 (unsigned long hwcap, const __ifunc_arg_t *features) { if (hwcap & HWCAP_USCAT) return true; @@ -75,6 +78,37 @@ ifunc1 (unsigned long hwcap, const __ifunc_arg_t *features) return false; } + +/* LSE128 atomic support encoded in ID_AA64ISAR0_EL1.Atomic, + bits[23:20]. The expected value is 0b0011. Check that. */ + +#define AT_FEAT_FIELD(isar0) (((isar0) >> 20) & 15) + +/* Ensure backwards compatibility with glibc <= 2.38. */ +#ifndef HWCAP2_LSE128 +#define HWCAP2_LSE128 (1UL << 47) +#endif + +static inline bool +has_lse128 (unsigned long hwcap, const __ifunc_arg_t *features) +{ + if (hwcap & _IFUNC_ARG_HWCAP + && features->_hwcap2 & HWCAP2_LSE128) + return true; + /* A 0 HWCAP2_LSE128 bit may be just as much a sign of missing HWCAP2 bit + support in older kernels as it is of CPU feature absence. Try fallback + method to guarantee LSE128 is not implemented. + + In the absence of HWCAP_CPUID, we are unable to check for LSE128. */ + if (!(hwcap & HWCAP_CPUID)) + return false; + unsigned long isar0; + asm volatile ("mrs %0, ID_AA64ISAR0_EL1" : "=r" (isar0)); + if (AT_FEAT_FIELD (isar0) >= 3) + return true; + return false; +} + #endif #include_next diff --git a/libatomic/configure b/libatomic/configure index d579bab96f8..8ab730d8082 100755 --- a/libatomic/configure +++ b/libatomic/configure @@ -656,6 +656,8 @@ LIBAT_BUILD_VERSIONED_SHLIB_FALSE LIBAT_BUILD_VERSIONED_SHLIB_TRUE OPT_LDFLAGS SECTION_LDFLAGS +ARCH_AARCH64_HAVE_LSE128_FALSE +ARCH_AARCH64_HAVE_LSE128_TRUE SYSROOT_CFLAGS_FOR_TARGET enable_aarch64_lse libtool_VERSION @@ -11456,7 +11458,7 @@ else lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2 lt_status=$lt_dlunknown cat > conftest.$ac_ext <<_LT_EOF -#line 11459 "configure" +#line 11461 "configure" #include "confdefs.h" #if HAVE_DLFCN_H @@ -11562,7 +11564,7 @@ else lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2 lt_status=$lt_dlunknown cat > conftest.$ac_ext <<_LT_EOF -#line 11565 "configure" +#line 11567 "configure" #include "confdefs.h" #if HAVE_DLFCN_H @@ -14697,6 +14699,57 @@ _ACEOF +# Check for target-specific assembly-level support for atomic operations. + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for armv9.4-a LSE128 insn support" >&5 +$as_echo_n "checking for armv9.4-a LSE128 insn support... " >&6; } +if ${libat_cv_have_feat_lse128+:} false; then : + $as_echo_n "(cached) " >&6 +else + + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +int +main () +{ +asm(".arch armv9-a+lse128") + ; + return 0; +} +_ACEOF + if { { eval echo "\"\$as_me\":${as_lineno-$LINENO}: \"$ac_compile\""; } >&5 + (eval $ac_compile) 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + eval libat_cv_have_feat_lse128=yes + else + eval libat_cv_have_feat_lse128=no + fi + rm -f conftest* + +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $libat_cv_have_feat_lse128" >&5 +$as_echo "$libat_cv_have_feat_lse128" >&6; } + + yesno=`echo $libat_cv_have_feat_lse128 | tr 'yesno' '1 0 '` + +cat >>confdefs.h <<_ACEOF +#define HAVE_FEAT_LSE128 $yesno +_ACEOF + + + if test x$libat_cv_have_feat_lse128 = xyes; then + ARCH_AARCH64_HAVE_LSE128_TRUE= + ARCH_AARCH64_HAVE_LSE128_FALSE='#' +else + ARCH_AARCH64_HAVE_LSE128_TRUE='#' + ARCH_AARCH64_HAVE_LSE128_FALSE= +fi + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether byte ordering is bigendian" >&5 $as_echo_n "checking whether byte ordering is bigendian... " >&6; } if ${ac_cv_c_bigendian+:} false; then : @@ -15989,6 +16042,10 @@ if test -z "${ENABLE_DARWIN_AT_RPATH_TRUE}" && test -z "${ENABLE_DARWIN_AT_RPATH as_fn_error $? "conditional \"ENABLE_DARWIN_AT_RPATH\" was never defined. Usually this means the macro was only invoked conditionally." "$LINENO" 5 fi +if test -z "${ARCH_AARCH64_HAVE_LSE128_TRUE}" && test -z "${ARCH_AARCH64_HAVE_LSE128_FALSE}"; then + as_fn_error $? "conditional \"ARCH_AARCH64_HAVE_LSE128\" was never defined. +Usually this means the macro was only invoked conditionally." "$LINENO" 5 +fi if test -z "${LIBAT_BUILD_VERSIONED_SHLIB_TRUE}" && test -z "${LIBAT_BUILD_VERSIONED_SHLIB_FALSE}"; then as_fn_error $? "conditional \"LIBAT_BUILD_VERSIONED_SHLIB\" was never defined. diff --git a/libatomic/configure.ac b/libatomic/configure.ac index 32a2cdb13ae..85824fa7614 100644 --- a/libatomic/configure.ac +++ b/libatomic/configure.ac @@ -206,6 +206,9 @@ LIBAT_FORALL_MODES([LIBAT_HAVE_ATOMIC_CAS]) LIBAT_FORALL_MODES([LIBAT_HAVE_ATOMIC_FETCH_ADD]) LIBAT_FORALL_MODES([LIBAT_HAVE_ATOMIC_FETCH_OP]) +# Check for target-specific assembly-level support for atomic operations. +LIBAT_TEST_FEAT_AARCH64_LSE128() + AC_C_BIGENDIAN # I don't like the default behaviour of WORDS_BIGENDIAN undefined for LE. AH_BOTTOM( -- 2.42.0