From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by sourceware.org (Postfix) with ESMTPS id 3F91B3858429 for ; Wed, 21 Feb 2024 09:58:31 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3F91B3858429 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=oracle.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=oracle.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 3F91B3858429 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=205.220.177.32 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1708509514; cv=pass; b=Ng9t1oei+IN5+2Gur8cYXnwZzKQ89KOk1QfxA7+Xdr4dJpF524sNQ24QvMEeLbSht59JL2QB+vjGJy73OWMe65DSRdMAWtyiFp7DbEAUiy4LFhjm25lY88Tnp4oabAt4xC6d/RRRtM83/NzvrezXJPXBpSOcthpX7Acycfdh1OQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1708509514; c=relaxed/simple; bh=d/2AknYTutf+Ct1NhqScabd8QQpbF8TspIWM59OHALQ=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=wajGExmySDfjPOXqvzJIupON/zrmRmHDPQywXmwR0EtMvIRyz7OTiBqjqGxefDl7W7rrpMHEdJD5+6g4YO4vNfPgT6jQALC2jLmubUQPGbITxEFWOTnf/XAGFOuH8X8YIhB49Z+norgH1BtTerUGLTmJ++wPlC9B8NqO5I8LefQ= ARC-Authentication-Results: i=2; server2.sourceware.org Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41L9hxLr020223 for ; Wed, 21 Feb 2024 09:58:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : in-reply-to : references : date : message-id : content-type : mime-version; s=corp-2023-11-20; bh=2H3IgHj7l9f5RaSjqckfJ3+tSjcg99W+shbhz9dgmxI=; b=a4A2tOSIV9+xlZIHWUZW3un8IqKpiJdc/9SfhL5yRinCWd8O3p2e4XlQd20kHZMeZWPk BlryWnu6POGz7J2t7+YwXfcBcL+qqQMKnKSXiLN1ojMJ8bA+Cbho8BIyXN6ZYE03uS96 WKM9r3iG8n7zeWHdjKmSctPBZSsW5E5yOVyFJrVkv/6hG5dsjiJ5n7Z/HTeikKZcB9ph NhZCFonkUjDU1QuRsYvCVT3rBO4dXJgtencqq5fseSvGjtqS7Wu/1Xaztx/M50wtI1+k bnk+hyqr4EG6tO4r+BydZAjO+JeR9OX6RlGIcs7vOKvLf7OggDbqNFrFQjKNcK6j12Fn 4w== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3wakk41e4k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 21 Feb 2024 09:58:30 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 41L8iXK9037793 for ; Wed, 21 Feb 2024 09:58:29 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3wak88qjpa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 21 Feb 2024 09:58:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nJ/ICsWHXgiaa3SO8W3W+xZDowFbXfA4k1h2qWtXtJn9LlY6DxLqT4nzDBU79N7d8K+pkqsc6SgBGhexAOxrzOFXKskqLmUL3lw+ZN20wcnxxQqVIUVFXJ5XXPID/zuzosPRseaNzDchjgmhzIJGToqVqRKTyIgXaaw4JIGrGd86lxj6YJd+IWJlfm4molZU955OUn7HrluJr4BDApK69qyp0grKcxJ8kYVcqptjl0zZJjcZHFSDBdGh8uPgPcURm7ti546cs7X8PFk0apF+6ZkO6m1P+X/fPcfYuLbG7XWjmLC41Zbqa8fxXwb/yLeHQ39qAYTlPmc3onLxouvZ7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2H3IgHj7l9f5RaSjqckfJ3+tSjcg99W+shbhz9dgmxI=; b=I8AqNRU0RYJu0vEFwUAUt6DK24PD20nQeD7id+xJMaW/9cyNf5ni181uvJO9ilxNKKcHi76SlIkuHQBQXTqcLzg2rbtCH+vKe8l3GCD4R4fDStL6DroHypyDXyzUbofaindZ3b/xf9Ca0TGxCZzlpa2rSqtJVQZ+cD8Aubjhb5BSXXYWqkjk22fWqKAaY+Z9XsMsBxIY32efeTmO4E2Cl65Dj04RpdmZr+/cVvSs4NUh2xZjgTtjEvIZ9eUizEUsCYrmZdf+S1BdxgnhXHL0YxxWPffidu088PXG+kjfE6fsmItkwnXTQBsFqk/7UAqJDoyCMlLZHdUoCrTyK4eyFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2H3IgHj7l9f5RaSjqckfJ3+tSjcg99W+shbhz9dgmxI=; b=ZgEBglSV3AP48C+kqMf7AXLj5TYSmumueLrydcP6wSvaAJNq4KPXZo9BAVmGD12ikvQuG0YAVXSTO5jk+h76+tJnrgoX2y031L5w7OEkThkCNlQ1aYHTowO7t65O9BKTgLyZ33xnXo0zZxhDBH05I3Drg/EgTp8LdQJXjdM4ynQ= Received: from DM6PR10MB3113.namprd10.prod.outlook.com (2603:10b6:5:1a7::12) by IA0PR10MB7622.namprd10.prod.outlook.com (2603:10b6:208:483::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.39; Wed, 21 Feb 2024 09:58:05 +0000 Received: from DM6PR10MB3113.namprd10.prod.outlook.com ([fe80::cafd:c8d6:e2c3:3a1]) by DM6PR10MB3113.namprd10.prod.outlook.com ([fe80::cafd:c8d6:e2c3:3a1%4]) with mapi id 15.20.7316.018; Wed, 21 Feb 2024 09:58:05 +0000 From: "Jose E. Marchesi" To: David Faust Cc: gcc-patches@gcc.gnu.org Subject: Re: [PATCH v3] bpf: add inline memmove and memcpy expansion In-Reply-To: <20240220225651.24206-1-david.faust@oracle.com> (David Faust's message of "Tue, 20 Feb 2024 14:56:51 -0800") References: <20240220225651.24206-1-david.faust@oracle.com> Date: Wed, 21 Feb 2024 10:57:58 +0100 Message-ID: <8734tm3zrt.fsf@oracle.com> User-Agent: Gnus/5.13 (Gnus v5.13) Content-Type: text/plain X-ClientProxiedBy: SG2PR06CA0242.apcprd06.prod.outlook.com (2603:1096:4:ac::26) To DM6PR10MB3113.namprd10.prod.outlook.com (2603:10b6:5:1a7::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB3113:EE_|IA0PR10MB7622:EE_ X-MS-Office365-Filtering-Correlation-Id: cca0b3a6-0f87-4a96-ef71-08dc32c39903 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: v6h/vm68thdHWtB2X7xmNESkuv+p5KmZ7638LKCBKBfubjpKulJwBVLbYc4f1yKkUhvJUxakkoS0bPrVc5GROXAQgbK8mIjrAeetvjkoAm3wbOig7g6SdcvoXO2y13PsxipJKsK9lHmenBtEkGzcTcks8m+5toyiaolP0UcAggvYRS18LfldlFiOc7GyXJXJoezm6h0fmXDZ8PPFDmCSYjMdagFEcQx62cZIeZrwAghEIs038NykdQvtddT/b3YKOJ2dspdQHSwKKuG3FsUl08ct1P6y5vO54xbfvoXsu3SU7mAN7ihxLOjXlxpX92iJ3bPIl9y55+sjIk9/S4kOs0d1kgXpbmLeEhHZK0/pFqU+tGN1K/JQbdarDCNyXdsEZ6X04rIRLOMIBomyudSKOPVRGGNIC6yu1oy8ndkJj28gFzPAQoNCRZ5fS+0eraFkz9v/avV6cz4ihUozR6oFcyWAEUkq3wYwP2ecjEZSGrJUEqeZdOUcDpPH5c5O1L3FrI6UPClFdPiRvHIQcxOr5aiO4NjG5sfv3+QrgdBybZmSu9yHhbWJA9YhNtQlzjuu X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB3113.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(230273577357003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?byrNmyN7KzMLl7ymMHCykk5S+fm0/Ve+M8uG085V/eQh5LjMrkVbD/3zjDwh?= =?us-ascii?Q?uLrShUXRKoclseBlpF6UGxwDUChUjBD3IDaiUkTWKTHwp5mq8Kqk8Qd5ps+k?= =?us-ascii?Q?JQqAUrCuZN7Pb40fvoYrE040n9d9il09q90vScsznfO4Q95VbEnpheasuOtS?= =?us-ascii?Q?YiS7d5TD+CjsXFhkAmuRwLSON6kZ0Hle+xtQqIZ6Y6EnOaD4o+rF9FI7civp?= =?us-ascii?Q?1tO3C14Fs2Yt80/q85lo7cdrybXs4f8KiYB3y0YnZTKvZXSwLq7x51slm91U?= =?us-ascii?Q?XTqLSxycuwOoCn7A2tSLL/U7lUXi9ZJFNN8bway9g+QUeK9hvnRQ/OY35OvR?= =?us-ascii?Q?MaXGhm+k6suSnroNR3e9vnMCpEqtSBOvYBqk85Xk7MKrmmyX9iiHtxzuIC+F?= =?us-ascii?Q?JXBDHQUI/oyJphrYL3mVkvGH+zQIf+ef/5FqT8JLsM2eDnOiX0QrOW6rD4ca?= =?us-ascii?Q?hJ87CsW6xTLOSWm6xaK4lU9cT29erL2oV/SnZ8d3NgiL13EQ6oRElbXssQOQ?= =?us-ascii?Q?yrdkhRUV+7obpGplzmkwLE1ckUjQRXGT2qwsFfn0JjDC5qG6riuulISbw3/S?= =?us-ascii?Q?REz2ffTpvUbkMpPTe6Xx82QDl9lOtTSClRz5MQ3pDYsS0Pfbto/j/4HvG7Ap?= =?us-ascii?Q?hsnvcgtGyRrsZoxYROC4cAhEPLpFri5Y8BbX/xSe91Ct3T2hvvCAok6/ZVYK?= =?us-ascii?Q?emd+tSulIv5Xpf22/Qd2Iflx8nb+LGf0SnzrdVGr1P73HRx27rUwk5EU/hPJ?= =?us-ascii?Q?pANWzCqVpZsmNT9D83YqSzGxUgYiJwyASYtHl2Mz5dhKcsAVYCv4zHxCK7en?= =?us-ascii?Q?PrpURsLloYfLG0wreFr4BgkDHPXuIr+QjlN9GrrShgMpEP2VqIJpSjGDITzX?= =?us-ascii?Q?+YmA/3xpf3OGg38I2/onFu0VehqPhJT+g5YWplDm4N2kgBFNrv7kfPNscWXM?= =?us-ascii?Q?JDJUAVzhlP1R4rl3GIXQ4PExMMxVcx7InJpQIeLv9t8JbpYYEcmhUgN2SSBq?= =?us-ascii?Q?dYcP4Eg8BMoHKG8wWCAAbL/DOxbduch9W7fCrBf9J2vYhQkmB18TFSmMh2wn?= =?us-ascii?Q?oG2W0V7JPzndpUE5GA36/2aYHN0zSS8PncHTdwe08GamSQ/JmsVwpdEcK0Pp?= =?us-ascii?Q?SLo4Dcw4DxtDMjfLOhRs16i5u0wf6N2G9Mf6d0OWjS5mC4Vcy9dziqfyzOM6?= =?us-ascii?Q?i5JgIutCO3m+SsQIRU0h4Xi7VeOS114SSsjzUVS62WNbmradARANGSViBloF?= =?us-ascii?Q?uOkm+j0EA6PtaPQbGAWahY5WLjVz9pFynJqXTSWDbfcs3Z+sZwyFcj6ZIRDW?= =?us-ascii?Q?hu3SbVXdQ3kBE/wKaF9wN/TFWdsz8Iftx9azIPTHVcy0PlF9nhHeQze4cwsb?= =?us-ascii?Q?ioTp2H29lgTyKsQfJRmwmGpw+T1tv3A+Asu1TXiZkoNBqTf3JA8erlnTExE1?= =?us-ascii?Q?UjXXOE+jQGokyWymTKW5THIGQ2/LhKcvH03ILhaBURXuH5qLasSJlev1IDXM?= =?us-ascii?Q?8EFXj522N6WNGuF2hxoIZ+7jtULp6Z+GI2ZMFgrK6xeQGK7yJvm4bPSee5sQ?= =?us-ascii?Q?0J5rUSjbuCxK8SV6RBdIu+lI09geeBjMAvAHI83192no/Br5MXD0aVnUFUA1?= =?us-ascii?Q?ig=3D=3D?= X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: ZESYX6wvVg1NhplOagc19AqH1xUvfRcT/UdgF8eA+7P1W7Agob1pyzjieFdn2AuPFM9gdlQFPP8CanEDg2yABz/iB+qISn89XsgDR81m+MwMGLRli6Cx64TBFaGH+vzsS8kL/cw3TFrWuJv0YMiF6cqPxW1fwvvdEYH7TtjcdvvqMKIz3phKAJVqAr9Bo0BC9TEFe6Iw8OvGLq9ZsHDtAuYxkcMoN5NVHT9fMOQpDqkZf6SV/srKh4ry3nds0AXe2dMaH2hs//NOiKfGDV80LHCN8FjFQ4v6vkXUr4u9Rchd/7yG0vdYIxElWUUD2X677Ypf0aBo7MknhhWghGtRNMy3wWHJjFkre8fqJllezqEvwK30nGNd4N5qEw0dD/iJs10QOGtgNSmJqJ5Pd0/l1T2AC2LLmDcZfBiosiYo4vW9guiLnE9ofgC6NvVFPsymhel41R3Wxzq1+7ChMV1cVKfCeOrwSwPWkYvXzRGHm78ZCxQYL30oz700V32gJ6Pv6ThH34EMBzaZK9unISA4Kp9u/DUIK8KRQIEWzd77zpLoFO33bXebvr8MyskP0AlpZaB/LokAJ5+7sTE7bRodZ9DZZqlAy2qxB5o92aOrcD8= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cca0b3a6-0f87-4a96-ef71-08dc32c39903 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB3113.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Feb 2024 09:58:05.0751 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: H2pkcwA7O7YqkbY+C2tslYOFRoS9Z3FSzq8oIvXkkBIS/qfMtyJEeZkrQrpXyGwy6wCmpBzW0rqNKu/7CqD/mRAmApvdkZcVfAmTHuEdl2Q= X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR10MB7622 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-20_06,2024-02-20_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402210078 X-Proofpoint-ORIG-GUID: E7CskjRPtcWmZByOuKrV6Lxd5AFl2PK3 X-Proofpoint-GUID: E7CskjRPtcWmZByOuKrV6Lxd5AFl2PK3 X-Spam-Status: No, score=-12.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,GIT_PATCH_0,KAM_SHORT,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: Hi David. This is OK. Thank you, very nice stuff! > [Changes from v2: > - Fix incorrectly passing a location instead of OPT_W* for warning (). > - Reword warning/error message and test accordingly. ] > > [Changes from v1: Jose's review comments, all of which I agree with. > - Fix 'implments' typo in commit message. > - Change check that alignment is CONST_INT to gcc_assert (). > - Change default case in alignment switch to gcc_unreachable (). > - Reword error message for non-constant size memmove/memcpy, and > update test for the error accordingly. > - Delete CPYMEM_EXPAND_ERR macro, since it was now only used in > one place. ] > > BPF programs are not typically linked, which means we cannot fall back > on library calls to implement __builtin_{memmove,memcpy} and should > always expand them inline if possible. > > GCC already successfully expands these builtins inline in many cases, > but failed to do so for a few for simple cases involving overlapping > memmove in the kernel BPF selftests and was instead emitting a libcall. > > This patch implements a simple inline expansion of memcpy and memmove in > the BPF backend in a verifier-friendly way, with the caveat that the > size must be an integer constant, which is also required by clang. > > Tested for bpf-unknown-none on x86_64-linux-gnu host. > > Also tested against the BPF verifier by compiling and loading a test > program with overlapping memmove (essentially the memmove-1.c test) > which failed before due to a libcall, and now successfully loads and > passes the verifier. > > gcc/ > > * config/bpf/bpf-protos.h (bpf_expand_cpymem): New. > * config/bpf/bpf.cc: (emit_move_loop, bpf_expand_cpymem): New. > * config/bpf/bpf.md: (cpymemdi, movmemdi): New define_expands. > > gcc/testsuite/ > > * gcc.target/bpf/memcpy-1.c: New test. > * gcc.target/bpf/memmove-1.c: New test. > * gcc.target/bpf/memmove-2.c: New test. > --- > gcc/config/bpf/bpf-protos.h | 2 + > gcc/config/bpf/bpf.cc | 115 +++++++++++++++++++++++ > gcc/config/bpf/bpf.md | 36 +++++++ > gcc/testsuite/gcc.target/bpf/memcpy-1.c | 26 +++++ > gcc/testsuite/gcc.target/bpf/memmove-1.c | 46 +++++++++ > gcc/testsuite/gcc.target/bpf/memmove-2.c | 23 +++++ > 6 files changed, 248 insertions(+) > create mode 100644 gcc/testsuite/gcc.target/bpf/memcpy-1.c > create mode 100644 gcc/testsuite/gcc.target/bpf/memmove-1.c > create mode 100644 gcc/testsuite/gcc.target/bpf/memmove-2.c > > diff --git a/gcc/config/bpf/bpf-protos.h b/gcc/config/bpf/bpf-protos.h > index 46d950bd990..366acb87ae4 100644 > --- a/gcc/config/bpf/bpf-protos.h > +++ b/gcc/config/bpf/bpf-protos.h > @@ -35,4 +35,6 @@ const char *bpf_add_core_reloc (rtx *operands, const char *templ); > class gimple_opt_pass; > gimple_opt_pass *make_pass_lower_bpf_core (gcc::context *ctxt); > > +bool bpf_expand_cpymem (rtx *, bool); > + > #endif /* ! GCC_BPF_PROTOS_H */ > diff --git a/gcc/config/bpf/bpf.cc b/gcc/config/bpf/bpf.cc > index d6ca47eeecb..f9ac263613a 100644 > --- a/gcc/config/bpf/bpf.cc > +++ b/gcc/config/bpf/bpf.cc > @@ -1184,6 +1184,121 @@ bpf_use_by_pieces_infrastructure_p (unsigned HOST_WIDE_INT size, > #define TARGET_USE_BY_PIECES_INFRASTRUCTURE_P \ > bpf_use_by_pieces_infrastructure_p > > +/* Helper for bpf_expand_cpymem. Emit an unrolled loop moving the bytes > + from SRC to DST. */ > + > +static void > +emit_move_loop (rtx src, rtx dst, machine_mode mode, int offset, int inc, > + unsigned iters, unsigned remainder) > +{ > + rtx reg = gen_reg_rtx (mode); > + > + /* First copy in chunks as large as alignment permits. */ > + for (unsigned int i = 0; i < iters; i++) > + { > + emit_move_insn (reg, adjust_address (src, mode, offset)); > + emit_move_insn (adjust_address (dst, mode, offset), reg); > + offset += inc; > + } > + > + /* Handle remaining bytes which might be smaller than the chunks > + used above. */ > + if (remainder & 4) > + { > + emit_move_insn (reg, adjust_address (src, SImode, offset)); > + emit_move_insn (adjust_address (dst, SImode, offset), reg); > + offset += (inc < 0 ? -4 : 4); > + remainder -= 4; > + } > + if (remainder & 2) > + { > + emit_move_insn (reg, adjust_address (src, HImode, offset)); > + emit_move_insn (adjust_address (dst, HImode, offset), reg); > + offset += (inc < 0 ? -2 : 2); > + remainder -= 2; > + } > + if (remainder & 1) > + { > + emit_move_insn (reg, adjust_address (src, QImode, offset)); > + emit_move_insn (adjust_address (dst, QImode, offset), reg); > + } > +} > + > +/* Expand cpymem/movmem, as from __builtin_memcpy/memmove. > + OPERANDS are the same as the cpymem/movmem patterns. > + IS_MOVE is true if this is a memmove, false for memcpy. > + Return true if we successfully expanded, or false if we cannot > + and must punt to a libcall. */ > + > +bool > +bpf_expand_cpymem (rtx *operands, bool is_move) > +{ > + /* Size must be constant for this expansion to work. */ > + if (!CONST_INT_P (operands[2])) > + { > + const char *name = is_move ? "memmove" : "memcpy"; > + if (flag_building_libgcc) > + warning (0, "could not inline call to %<__builtin_%s%>: " > + "size must be constant", name); > + else > + error ("could not inline call to %<__builtin_%s%>: " > + "size must be constant", name); > + return false; > + } > + > + /* Alignment is a CONST_INT. */ > + gcc_assert (CONST_INT_P (operands[3])); > + > + rtx dst = operands[0]; > + rtx src = operands[1]; > + rtx size = operands[2]; > + unsigned HOST_WIDE_INT size_bytes = UINTVAL (size); > + unsigned align = UINTVAL (operands[3]); > + enum machine_mode mode; > + switch (align) > + { > + case 1: mode = QImode; break; > + case 2: mode = HImode; break; > + case 4: mode = SImode; break; > + case 8: mode = DImode; break; > + default: > + gcc_unreachable (); > + } > + > + unsigned iters = size_bytes >> ceil_log2 (align); > + unsigned remainder = size_bytes & (align - 1); > + > + int inc = GET_MODE_SIZE (mode); > + rtx_code_label *fwd_label, *done_label; > + if (is_move) > + { > + /* For memmove, be careful of overlap. It is not a concern for memcpy. > + To handle overlap, we check (at runtime) if SRC < DST, and if so do > + the move "backwards" starting from SRC + SIZE. */ > + fwd_label = gen_label_rtx (); > + done_label = gen_label_rtx (); > + > + rtx dst_addr = copy_to_mode_reg (Pmode, XEXP (dst, 0)); > + rtx src_addr = copy_to_mode_reg (Pmode, XEXP (src, 0)); > + emit_cmp_and_jump_insns (src_addr, dst_addr, GEU, NULL_RTX, Pmode, > + true, fwd_label, profile_probability::even ()); > + > + /* Emit the "backwards" unrolled loop. */ > + emit_move_loop (src, dst, mode, size_bytes, -inc, iters, remainder); > + emit_jump_insn (gen_jump (done_label)); > + emit_barrier (); > + > + emit_label (fwd_label); > + } > + > + emit_move_loop (src, dst, mode, 0, inc, iters, remainder); > + > + if (is_move) > + emit_label (done_label); > + > + return true; > +} > + > /* Finally, build the GCC target. */ > > struct gcc_target targetm = TARGET_INITIALIZER; > diff --git a/gcc/config/bpf/bpf.md b/gcc/config/bpf/bpf.md > index 50df1aaa3e2..ca677bc6b50 100644 > --- a/gcc/config/bpf/bpf.md > +++ b/gcc/config/bpf/bpf.md > @@ -627,4 +627,40 @@ (define_insn "ldabs" > "{ldabs\t%0|r0 = *( *) skb[%0]}" > [(set_attr "type" "ld")]) > > +;;; memmove and memcopy > + > +;; 0 is dst > +;; 1 is src > +;; 2 is size of copy in bytes > +;; 3 is alignment > + > +(define_expand "cpymemdi" > + [(match_operand:BLK 0 "memory_operand") > + (match_operand:BLK 1 "memory_operand") > + (match_operand:DI 2 "general_operand") > + (match_operand:DI 3 "immediate_operand")] > + "" > +{ > + if (bpf_expand_cpymem (operands, false)) > + DONE; > + FAIL; > +}) > + > +;; 0 is dst > +;; 1 is src > +;; 2 is size of copy in bytes > +;; 3 is alignment > + > +(define_expand "movmemdi" > + [(match_operand:BLK 0 "memory_operand") > + (match_operand:BLK 1 "memory_operand") > + (match_operand:DI 2 "general_operand") > + (match_operand:DI 3 "immediate_operand")] > + "" > +{ > + if (bpf_expand_cpymem (operands, true)) > + DONE; > + FAIL; > +}) > + > (include "atomic.md") > diff --git a/gcc/testsuite/gcc.target/bpf/memcpy-1.c b/gcc/testsuite/gcc.target/bpf/memcpy-1.c > new file mode 100644 > index 00000000000..6c9707f24e8 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/bpf/memcpy-1.c > @@ -0,0 +1,26 @@ > +/* Ensure memcpy is expanded inline rather than emitting a libcall. */ > + > +/* { dg-do compile } */ > +/* { dg-options "-O2" } */ > + > +struct context { > + unsigned int data; > + unsigned int data_end; > + unsigned int data_meta; > + unsigned int ingress; > + unsigned int queue_index; > + unsigned int egress; > +}; > + > +void > +cpy_1(struct context *ctx) > +{ > + void *data = (void *)(long)ctx->data; > + char *dest; > + dest = data; > + dest += 16; > + > + __builtin_memcpy (dest, data, 8); > +} > + > +/* { dg-final { scan-assembler-times "call" 0 } } */ > diff --git a/gcc/testsuite/gcc.target/bpf/memmove-1.c b/gcc/testsuite/gcc.target/bpf/memmove-1.c > new file mode 100644 > index 00000000000..3b8ba82639e > --- /dev/null > +++ b/gcc/testsuite/gcc.target/bpf/memmove-1.c > @@ -0,0 +1,46 @@ > +/* Ensure memmove is expanded inline rather than emitting a libcall. */ > + > +/* { dg-do compile } */ > +/* { dg-options "-O2" } */ > + > +struct context { > + unsigned int data; > + unsigned int data_end; > + unsigned int data_meta; > + unsigned int ingress; > + unsigned int queue_index; > + unsigned int egress; > +}; > + > +void > +mov_1_nooverlap (struct context *ctx) > +{ > + void *data = (void *)(long)ctx->data; > + char *dest; > + dest = data; > + dest += 16; > + > + __builtin_memmove (dest, data, 12); > +} > + > +void > +mov_1_overlap (struct context *ctx) > +{ > + void *data = (void *)(long)ctx->data; > + char *dest; > + dest = data; > + dest += 4; > + > + __builtin_memmove (dest, data, 12); > +} > + > +void > +mov_1_arbitrary (struct context *ctx_a, struct context *ctx_b) > +{ > + void *src = (void *)(long)ctx_a->data; > + void *dst = (void *)(long)ctx_b->data; > + > + __builtin_memmove (dst, src, 12); > +} > + > +/* { dg-final { scan-assembler-times "call" 0 } } */ > diff --git a/gcc/testsuite/gcc.target/bpf/memmove-2.c b/gcc/testsuite/gcc.target/bpf/memmove-2.c > new file mode 100644 > index 00000000000..c0f92d40d92 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/bpf/memmove-2.c > @@ -0,0 +1,23 @@ > +/* Test that we error if memmove cannot be expanded inline. */ > + > +/* { dg-do compile } */ > +/* { dg-options "-O2" } */ > + > +typedef unsigned int __u32; > + > +struct context { > + unsigned int data; > + unsigned int data_end; > + unsigned int data_meta; > +}; > + > +void > +mov_2_unsupported (struct context *ctx) > +{ > + void *data = (void *)(long)ctx->data; > + char *dest; > + dest = data; > + dest += 4; > + > + __builtin_memmove (dest, data, ctx->data_meta); /* { dg-error "could not inline call" } */ > +}