From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by sourceware.org (Postfix) with ESMTPS id 1F1793860006 for ; Wed, 14 Feb 2024 14:33:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1F1793860006 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1F1793860006 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::52c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1707921186; cv=none; b=n4DO5Y6QxOtYs1wJ8JpZmRX+KY7MGlyY+SA/gMrpg2Cr70+0wIlYljaS/io2ypUT0uE5lnsWSH6u0D/5QePuAnk7mofnVLRRP+4H51WbZF4m/cXZrzVle++EmmYeSQtXJ2nt5rYQX9G0IhJnemdL9uFmnXl95Zxuila3ghgwPrU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1707921186; c=relaxed/simple; bh=RQeCsTdLzL95jS0ANE2Z4YVo2AodGkOejdkJeicqBls=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=sRmi1YtWFc3FXeGIhwWRn2i3C2qRCqxDPD0pBkusPqrtqEFStsLLEsphbcsK4+KWlb0vrVwKEIyiVKAOCvfvN4KdfyPWy3zr2FD00NffH7iaAn1PmjMwSqoXTOSt63PwZPcRhn3BfNJXL11ajRuQpPhuK9p7gjaSIQHjJdOgagk= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-5d8ddbac4fbso4781630a12.0 for ; Wed, 14 Feb 2024 06:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1707921181; x=1708525981; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d0YVHUk/lddbP97jJ8T2SQlEKI7riLRj+3PwAqGif+0=; b=mSDcxn9Pr5TerLMEqM/8Rd2/LSr9YpQlDAiq4PsUWiAMB5gp4mfLtc1s4ZPGG2FOFD 3KlkN2PvJUro2khXs2NTam41h63OEpDQ6j5KRv6dlkNneZVR2ZsM3/fjau7lY9AtcfWa MNOF69amM29RiatWEAv6QvPl1Yfq2hiPQOdiuHnD3QnpOuR4qUXI4CzKgP5V9sg7fmip FZrHMGTFbx5ebd3kPVt6KpRx6d9F53D0yfCAMkVJ4rBtI5CMBXv7fnOFEvKM/yBB2mLj CuRhA9RMRlGU7VHcjHNT7lDO1EStW62PFxPRSJQOSKxTy9UuNXETEg4ullEIlj4MfT7A P1Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707921181; x=1708525981; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d0YVHUk/lddbP97jJ8T2SQlEKI7riLRj+3PwAqGif+0=; b=K24Wj6wcN8Sis33+DzNM+IcBkb/NyGftaZ4dS07b5EZSx4imYapVAiMIBTGHQCb2H5 ankM0MIAP57mF+eWr6OKX1ywdeDS8IMPpOeCItTX7/rWsEjhmHR4we08dhweX1RASQcT wxOumUrI7YtiZn72eHFYp1uLaC+Gi1ECr1ShKb8y8DuviVk8w+jIA2Z3/ZsDl4MQpcQg cO7+j1Uk0nuU1K/72UhJ6uVav9LhzwcFemBqvsxGkfLBpDPSE89QMsQe1iDNakJsRU1Y 1lt+wa+fXQjEsxZ6Mcyo9iCrxfbOOxrvKfPJ4MKX+LPxWrm3UuZNHoZzctYnXvFL+g2v VJaA== X-Gm-Message-State: AOJu0Yx4UmAjpvkb8y/xp7HdUd2T7dBhoz2jrY/tSd04PLepyAhZqoVg 6uJGKByQD9KAnib0czWuATXibzqfNbesyfkhr9+PBOO0Mw/M+mOwnRBpEdJe+j21UGQ1SCV1jRb E X-Google-Smtp-Source: AGHT+IFzzNZL1XPE9/kAZMz4/GPSFCyGsUMYMfYcAE57C3t8E7z48d2+4YZkKMcuQEv0IKTpLqJX8g== X-Received: by 2002:a17:90b:450b:b0:298:bca7:c530 with SMTP id iu11-20020a17090b450b00b00298bca7c530mr2622850pjb.42.1707921181157; Wed, 14 Feb 2024 06:33:01 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCVmCbKrBganTHqPp3CdP5eN0uiq8UZQLdFfg6rqSWesgabaM3qJo8WhsLwpxkVsPKoBOGsnZbcAt+RNwnALQUb5fF/EbGOJUT4QVMdDD6r+9nuUsNolPqiTnQwceRqYjHGaFIvKeYYntMTIodg36/lQuFM03GT6wqzn/VwQB8xs Received: from evan.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id k21-20020a635a55000000b005dc98d9114bsm1140255pgm.43.2024.02.14.06.33.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 06:33:00 -0800 (PST) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, Florian Weimer , slewis@rivosinc.com, palmer@rivosinc.com, Evan Green Subject: [PATCH v12 7/7] riscv: Add and use alignment-ignorant memcpy Date: Wed, 14 Feb 2024 06:31:58 -0800 Message-Id: <20240214143159.2951158-8-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240214143159.2951158-1-evan@rivosinc.com> References: <20240214143159.2951158-1-evan@rivosinc.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,GIT_PATCH_0,KAM_SHORT,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: For CPU implementations that can perform unaligned accesses with little or no performance penalty, create a memcpy implementation that does not bother aligning buffers. It will use a block of integer registers, a single integer register, and fall back to bytewise copy for the remainder. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- (no changes since v11) Changes in v11: - Update copyrights to 2024 (Adhemerval) - Avoid implicit check in select_memcpy_ifunc (Adhemerval) - Remove hidden_def (__memcpy_noalignment) (Adhemerval) Changes in v10: - One line per function in Makefile for memcpy (Adhemerval) - Space before argument-like things (Adhemerval) Changes in v7: - Use new helper function in memcpy ifunc selector (Richard) Changes in v6: - Fix a couple regressions in the assembly from v5 :/ - Use passed hwprobe pointer in memcpy ifunc selector. Changes in v5: - Do unaligned word access for final trailing bytes (Richard) Changes in v4: - Fixed comment style (Florian) Changes in v3: - Word align dest for large memcpy()s. - Add tags - Remove spurious blank line from sysdeps/riscv/memcpy.c Changes in v2: - Used _MASK instead of _FAST value itself. --- sysdeps/riscv/memcopy.h | 26 ++++ sysdeps/riscv/memcpy.c | 63 ++++++++ sysdeps/riscv/memcpy_noalignment.S | 136 ++++++++++++++++++ sysdeps/unix/sysv/linux/riscv/Makefile | 9 ++ .../unix/sysv/linux/riscv/memcpy-generic.c | 24 ++++ 5 files changed, 258 insertions(+) create mode 100644 sysdeps/riscv/memcopy.h create mode 100644 sysdeps/riscv/memcpy.c create mode 100644 sysdeps/riscv/memcpy_noalignment.S create mode 100644 sysdeps/unix/sysv/linux/riscv/memcpy-generic.c diff --git a/sysdeps/riscv/memcopy.h b/sysdeps/riscv/memcopy.h new file mode 100644 index 0000000000..27675964b0 --- /dev/null +++ b/sysdeps/riscv/memcopy.h @@ -0,0 +1,26 @@ +/* memcopy.h -- definitions for memory copy functions. RISC-V version. + Copyright (C) 2024 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +/* Redefine the generic memcpy implementation to __memcpy_generic, so + the memcpy ifunc can select between generic and special versions. + In rtld, don't bother with all the ifunciness. */ +#if IS_IN (libc) +#define MEMCPY __memcpy_generic +#endif diff --git a/sysdeps/riscv/memcpy.c b/sysdeps/riscv/memcpy.c new file mode 100644 index 0000000000..20f9548c44 --- /dev/null +++ b/sysdeps/riscv/memcpy.c @@ -0,0 +1,63 @@ +/* Multiple versions of memcpy. + All versions must be listed in ifunc-impl-list.c. + Copyright (C) 2017-2024 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Redefine memcpy so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef memcpy +# define memcpy __redirect_memcpy +# include +# include +# include +# include +# include + +# define INIT_ARCH() + +extern __typeof (__redirect_memcpy) __libc_memcpy; + +extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_noalignment attribute_hidden; + +static inline __typeof (__redirect_memcpy) * +select_memcpy_ifunc (uint64_t dl_hwcap, __riscv_hwprobe_t hwprobe_func) +{ + unsigned long long int value; + + INIT_ARCH (); + + if (__riscv_hwprobe_one (hwprobe_func, RISCV_HWPROBE_KEY_CPUPERF_0, &value) != 0) + return __memcpy_generic; + + if ((value & RISCV_HWPROBE_MISALIGNED_MASK) == RISCV_HWPROBE_MISALIGNED_FAST) + return __memcpy_noalignment; + + return __memcpy_generic; +} + +riscv_libc_ifunc (__libc_memcpy, select_memcpy_ifunc); + +# undef memcpy +strong_alias (__libc_memcpy, memcpy); +# ifdef SHARED +__hidden_ver1 (memcpy, __GI_memcpy, __redirect_memcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcpy); +# endif + +#endif diff --git a/sysdeps/riscv/memcpy_noalignment.S b/sysdeps/riscv/memcpy_noalignment.S new file mode 100644 index 0000000000..621f8d028f --- /dev/null +++ b/sysdeps/riscv/memcpy_noalignment.S @@ -0,0 +1,136 @@ +/* memcpy for RISC-V, ignoring buffer alignment + Copyright (C) 2024 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include + +/* void *memcpy(void *, const void *, size_t) */ +ENTRY (__memcpy_noalignment) + move t6, a0 /* Preserve return value */ + + /* Bail if 0 */ + beqz a2, 7f + + /* Jump to byte copy if size < SZREG */ + li a4, SZREG + bltu a2, a4, 5f + + /* Round down to the nearest "page" size */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + add a3, a1, a4 + + /* Copy the first word to get dest word aligned */ + andi a5, t6, SZREG-1 + beqz a5, 1f + REG_L a6, (a1) + REG_S a6, (t6) + + /* Align dst up to a word, move src and size as well. */ + addi t6, t6, SZREG-1 + andi t6, t6, ~(SZREG-1) + sub a5, t6, a0 + add a1, a1, a5 + sub a2, a2, a5 + + /* Recompute page count */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + +1: + /* Copy "pages" (chunks of 16 registers) */ + REG_L a4, 0(a1) + REG_L a5, SZREG(a1) + REG_L a6, 2*SZREG(a1) + REG_L a7, 3*SZREG(a1) + REG_L t0, 4*SZREG(a1) + REG_L t1, 5*SZREG(a1) + REG_L t2, 6*SZREG(a1) + REG_L t3, 7*SZREG(a1) + REG_L t4, 8*SZREG(a1) + REG_L t5, 9*SZREG(a1) + REG_S a4, 0(t6) + REG_S a5, SZREG(t6) + REG_S a6, 2*SZREG(t6) + REG_S a7, 3*SZREG(t6) + REG_S t0, 4*SZREG(t6) + REG_S t1, 5*SZREG(t6) + REG_S t2, 6*SZREG(t6) + REG_S t3, 7*SZREG(t6) + REG_S t4, 8*SZREG(t6) + REG_S t5, 9*SZREG(t6) + REG_L a4, 10*SZREG(a1) + REG_L a5, 11*SZREG(a1) + REG_L a6, 12*SZREG(a1) + REG_L a7, 13*SZREG(a1) + REG_L t0, 14*SZREG(a1) + REG_L t1, 15*SZREG(a1) + addi a1, a1, 16*SZREG + REG_S a4, 10*SZREG(t6) + REG_S a5, 11*SZREG(t6) + REG_S a6, 12*SZREG(t6) + REG_S a7, 13*SZREG(t6) + REG_S t0, 14*SZREG(t6) + REG_S t1, 15*SZREG(t6) + addi t6, t6, 16*SZREG + bltu a1, a3, 1b + andi a2, a2, (16*SZREG)-1 /* Update count */ + +2: + /* Remainder is smaller than a page, compute native word count */ + beqz a2, 7f + andi a5, a2, ~(SZREG-1) + andi a2, a2, (SZREG-1) + add a3, a1, a5 + /* Jump directly to last word if no words. */ + beqz a5, 4f + +3: + /* Use single native register copy */ + REG_L a4, 0(a1) + addi a1, a1, SZREG + REG_S a4, 0(t6) + addi t6, t6, SZREG + bltu a1, a3, 3b + + /* Jump directly out if no more bytes */ + beqz a2, 7f + +4: + /* Copy the last word unaligned */ + add a3, a1, a2 + add a4, t6, a2 + REG_L a5, -SZREG(a3) + REG_S a5, -SZREG(a4) + ret + +5: + /* Copy bytes when the total copy is . */ + +#include + +extern __typeof (memcpy) __memcpy_generic; +hidden_proto (__memcpy_generic) + +#include -- 2.34.1