From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2062.outbound.protection.outlook.com [40.107.94.62]) by sourceware.org (Postfix) with ESMTPS id 91E32396ECA3 for ; Thu, 7 Jan 2021 16:22:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 91E32396ECA3 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nYSxd0ppWTcahDCNo3yeOzHhgVsBLvMxHo6cki2onY4zYMbFHBzhpG60BG0SpT3iCBS5b7s50HE9gqOO71aTCEYBV/6hoTdjn8KrPxu7ZnPcj/fiw0lLU921Nj9DRTBO+KP5Mcbp0f1z2KN0rK9Hv968bm+9XozrMC5gLWy9LfNB441dILcKYH/MfvVI35OlHmVLTgRsI2+Z+Bn4P9jCvMi6Yl5woL08ClL3gAZsSh9Z6ija871zDXwy4KQOba/yq2t4EEJNeMWxMGoimcuWo8EalnG+22eoFoDvwRjceUw8CLZVOWNWtwFWNzqw0h8sQyAuP9PEqB4B5zRfHbUbyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=am3ux1nS9/khfUB0uLrEOVQ7s5cA00rG7qIigGYTRHE=; b=JJVylXKt2t9yFNle1b1HX3JpKCHK0zib7NSSYTQVinNvpB0IDYZdeh195PnijjC2krb6Jp5+XR78jugrNTTLyWeHuaV8kj2saMW46ei39UYNVYxU3H71rCU9GtiXJhs5SauhkZLklvsvVo86rl+yc8mo8Gr9q74tTvlJ3sapfCKnDW1UxbvyjZOb7S43r9xzy6m1xNbht7LLwwUDvV22EPQmzn4dnp/gZy415ncnbEAdnD/3Jl8qpXpT0OzeZiiXYHuQYl6S7UBB/AgktHf5ok2uSGPfccHG/4Z3AMokiX+zCCFF/E/4NZE94/EZSCcDYDRhFPYzYWJTUmR8pVY5nQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none Received: from BN6PR1201MB0081.namprd12.prod.outlook.com (2603:10b6:405:4f::12) by BN8PR12MB2948.namprd12.prod.outlook.com (2603:10b6:408:6d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3721.21; Thu, 7 Jan 2021 16:22:41 +0000 Received: from BN6PR1201MB0081.namprd12.prod.outlook.com ([fe80::20fa:a1a2:9e7d:1370]) by BN6PR1201MB0081.namprd12.prod.outlook.com ([fe80::20fa:a1a2:9e7d:1370%11]) with mapi id 15.20.3721.024; Thu, 7 Jan 2021 16:22:41 +0000 From: sajan.karumanchi@amd.com To: libc-alpha@sourceware.org, carlos@redhat.com, fweimer@redhat.com, hjl.tools@gmail.com Cc: Sajan Karumanchi , premachandra.mallappa@amd.com Subject: [PATCH] x86: Adding an upper bound for Enhanced REP MOVSB. Date: Thu, 7 Jan 2021 21:52:09 +0530 Message-Id: <20210107162209.4047176-1-sajan.karumanchi@amd.com> X-Mailer: git-send-email 2.25.1 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [165.204.156.251] X-ClientProxiedBy: MA1PR01CA0144.INDPRD01.PROD.OUTLOOK.COM (2603:1096:a00:71::14) To BN6PR1201MB0081.namprd12.prod.outlook.com (2603:10b6:405:4f::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from amd.com (165.204.156.251) by MA1PR01CA0144.INDPRD01.PROD.OUTLOOK.COM (2603:1096:a00:71::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3742.6 via Frontend Transport; Thu, 7 Jan 2021 16:22:39 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 15fb9bf5-0092-4750-63d9-08d8b32874fc X-MS-TrafficTypeDiagnostic: BN8PR12MB2948: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /72pwy9x4GzjkzkbwGb+htgS1gFTRyAepCzlGuQHuT287Ma918g85a+yGcx+AIiQySEhGpYS7fpMGv+AGwbVatMHb+cvrg/k2Oj4CV4TIQipxWxaq677+DkAyvHHqV8SwqxJoQGpzC8Mz6XQUX7ZhXUiKJNAtsmobhFIobGL97WsTnVAfnVIj91v/gHYiT649rGtymFkO6a05jqP8l9nihDm2m4GeP/4zaLGjZE1TYFBvBjpXjNop/1Upg34sIUt4JCgbphAt/bMeMpXCIjmgf+1vzRd16NW6jLlf/pOKKLD/DMk4p5d8jwM5hQw3w8Pn5j32b9x8sfJp/hP7CnS3L00TYnpZU9hlssFW+38wgNG85yLFCcXPQ8iHOiHBZfRL4szRz+VPjDF3GxUUP0o/vkFt0PFOmgOSxe/XbloCAi1qMdpOpEB/arpmGtQShXO X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR1201MB0081.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(346002)(396003)(136003)(39860400002)(376002)(2906002)(36756003)(186003)(8886007)(316002)(7696005)(26005)(66556008)(9686003)(86362001)(478600001)(66946007)(66476007)(16526019)(2616005)(52116002)(956004)(6666004)(55016002)(4326008)(1076003)(5660300002)(83380400001)(8676002)(8936002)(2004002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?qqTdO1H01AfNX5bYtlPZC9Lmuo+o8hgt6lm4cC8KwIrfY/ndj4Pc1ypzoakz?= =?us-ascii?Q?PVU+in08tM4edZr3j1wwYov2AYu83DtxRgo4NiiZLHIEU/FBtB125uhZ31V6?= =?us-ascii?Q?7NuFOyXO3FHnaWtHIXg8uGG7/XJBRMLzOW5h13IPKo+kULdhG346Vup6NoHR?= =?us-ascii?Q?WvXpf/kCGeUxinisOkdwmeExbXT46QkZvRp17MAjYxzTHu6ldLGR210mawji?= =?us-ascii?Q?ir6K/zqXrMXmuOWAFpgkitle33iSxXv2OeOkH1iFTtk5EogJGaYxF8mCtnof?= =?us-ascii?Q?JLe4wfYu9J/FxgadbN4mmOlBfSnrLu0RAfKtGbAzvtfk7ouYvHidL4CdXNiq?= =?us-ascii?Q?1TuCc5wM2XXAn/fGnY5R3V2N37lzdNSnQ1NgrlfCVhoGS7YfFaZnX+gBZJcn?= =?us-ascii?Q?2fsTEFR9WYujvPc/C6QbZDBvRyxqKZy8AkKs11+OzM/NlX0S4d8piS2PWmvy?= =?us-ascii?Q?nzt6GILexSuaQqYX3Ctuq8e8qQ9pRoND4sFO5eF4WrXFu6qBdaseV+MEMngd?= =?us-ascii?Q?KH9aaP3IW/t7lUz5TF13zLni46xpO3LuUdl99cBuwmrSuf8kgntkGLU5+F4l?= =?us-ascii?Q?751cWtkkr2ADdx9/uPXmXQYxz+zJ6VvCt42poCwHMYGZVdJqu2OR2ZuO8BzH?= =?us-ascii?Q?q1nggs7gUN0cwafrONrTfXAW9u3OxFSWY4pK1jVW69hi9HGRUq2Vq9SYkxve?= =?us-ascii?Q?j/ReJNwfLExIInFi2pWZxV35ysTGfYgvcmvpEbGf+l5cSzMCdUABkMOlndms?= =?us-ascii?Q?G/z+KKEuLMLQRqwNBaqdJi58uWiW9FuswZJmgLJv3t7zZxsHN7EjE48FDAmR?= =?us-ascii?Q?DGCLSF3clW1aXu3QPGeXaUA7QetW2CW1hUzccJ1nhWGqccWXrWA+ZQNKb4XE?= =?us-ascii?Q?iaQm05NRHNtd9bJ/v1V6JItYY5CJBkiLxw9cJO8JQFMVQbTN1WdJNgcxYXMS?= =?us-ascii?Q?teUeYJ8Ak/Aik1gEafULtjhwtEfppsAeToLJQcPgb87kFJHvr77fCrqPczY/?= =?us-ascii?Q?mjUM?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-AuthSource: BN6PR1201MB0081.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jan 2021 16:22:41.4667 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-Network-Message-Id: 15fb9bf5-0092-4750-63d9-08d8b32874fc X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: S2izEKBP0+/iJaz2F8Hb1I/omXAT547zIRSuh/DLf8yi22PhTtKqc9APSE0mWTbTDxNhKCTIySIHcw9t6MsPFg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB2948 X-Spam-Status: No, score=-14.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jan 2021 16:22:48 -0000 From: Sajan Karumanchi In the process of optimizing memcpy for AMD machines, we have found the vector move operations are outperforming enhanced REP MOVSB for data transfers above the L2 cache size on Zen3 architectures. To handle this use case, we are adding an upper bound parameter on enhanced REP MOVSB:'__x86_max_rep_movsb_threshold'. As per large-bench results, we are configuring this parameter to the L2 cache size for AMD machines and applicable from Zen3 architecture supporting the ERMS feature. For architectures other than AMD, it is the computed value of non-temporal threshold parameter. Reviewed-by: Premachandra Mallappa --- sysdeps/x86/cacheinfo.c | 15 ++++++++++++++- .../x86_64/multiarch/memmove-vec-unaligned-erms.S | 4 ++-- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/sysdeps/x86/cacheinfo.c b/sysdeps/x86/cacheinfo.c index 3fb4a028d8..9d7f8992be 100644 --- a/sysdeps/x86/cacheinfo.c +++ b/sysdeps/x86/cacheinfo.c @@ -1,5 +1,5 @@ /* x86_64 cache info. - Copyright (C) 2003-2020 Free Software Foundation, Inc. + Copyright (C) 2003-2021 Free Software Foundation, Inc. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -533,6 +533,9 @@ long int __x86_shared_non_temporal_threshold attribute_hidden; /* Threshold to use Enhanced REP MOVSB. */ long int __x86_rep_movsb_threshold attribute_hidden = 2048; +/* Threshold to stop using Enhanced REP MOVSB. */ +long int __x86_max_rep_movsb_threshold attribute_hidden = 512 * 1024; + /* Threshold to use Enhanced REP STOSB. */ long int __x86_rep_stosb_threshold attribute_hidden = 2048; @@ -839,6 +842,11 @@ init_cacheinfo (void) /* Account for exclusive L2 and L3 caches. */ shared += core; } + /* ERMS feature is implemented from Zen3 architecture and it is + performing poorly for data above L2 cache size. Henceforth, adding + an upper bound threshold parameter to limit the usage of Enhanced + REP MOVSB operations and setting its value to L2 cache size. */ + __x86_max_rep_movsb_threshold = core; } } @@ -909,6 +917,11 @@ init_cacheinfo (void) else __x86_rep_movsb_threshold = rep_movsb_threshold; + /* Setting the upper bound of ERMS to the known default value of + non-temporal threshold for architectures other than AMD. */ + if (cpu_features->basic.kind != arch_kind_amd) + __x86_max_rep_movsb_threshold = __x86_shared_non_temporal_threshold; + # if HAVE_TUNABLES __x86_rep_stosb_threshold = cpu_features->rep_stosb_threshold; # endif diff --git a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S index bd5dc1a3f3..c18eaf7ef6 100644 --- a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S @@ -1,5 +1,5 @@ /* memmove/memcpy/mempcpy with unaligned load/store and rep movsb - Copyright (C) 2016-2020 Free Software Foundation, Inc. + Copyright (C) 2016-2021 Free Software Foundation, Inc. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -233,7 +233,7 @@ L(return): ret L(movsb): - cmp __x86_shared_non_temporal_threshold(%rip), %RDX_LP + cmp __x86_max_rep_movsb_threshold(%rip), %RDX_LP jae L(more_8x_vec) cmpq %rsi, %rdi jb 1f -- 2.25.1