From: Michael Jeanson <mjeanson@efficios.com>
To: libc-alpha@sourceware.org
Cc: Michael Jeanson <mjeanson@efficios.com>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Subject: [PATCH v8 8/8] Linux: Use rseq to accelerate getcpu
Date: Tue, 6 Feb 2024 11:28:01 -0500 [thread overview]
Message-ID: <20240206162801.882585-9-mjeanson@efficios.com> (raw)
In-Reply-To: <20240206162801.882585-1-mjeanson@efficios.com>
On architectures that implement rseq_load32_load32_relaxed() (and thus
define RSEQ_HAS_LOAD32_LOAD32_RELAXED), when the node_id feature is
available, use rseq to fetch the cpu_id and node_id atomically with
respect to preemption and signal delivery to speed up getcpu() compared
to a vsyscall or system call implementation.
Loading both cpu_id and node_id atomically with respect to preemption
is required to ensure consistency of the topology mapping between cpu_id
and node_id due to migration between both loads.
On an aarch64 system (Snapdragon 8cx Gen 3) which lacks a vDSO for
getcpu() we measured an improvement from 130 ns to 1 ns while on x86_64
(i7-8550U) which has a vDSO we measured a more modest improvement from
10 ns to 2 ns.
Co-authored-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
---
sysdeps/unix/sysv/linux/getcpu.c | 32 ++++++++++++++++++++++++++++++--
1 file changed, 30 insertions(+), 2 deletions(-)
diff --git a/sysdeps/unix/sysv/linux/getcpu.c b/sysdeps/unix/sysv/linux/getcpu.c
index 0e7c3238c9..7e34d6d1eb 100644
--- a/sysdeps/unix/sysv/linux/getcpu.c
+++ b/sysdeps/unix/sysv/linux/getcpu.c
@@ -19,9 +19,10 @@
#include <sched.h>
#include <sysdep.h>
#include <sysdep-vdso.h>
+#include <rseq-internal.h>
-int
-__getcpu (unsigned int *cpu, unsigned int *node)
+static int
+vsyscall_getcpu (unsigned int *cpu, unsigned int *node)
{
#ifdef HAVE_GETCPU_VSYSCALL
return INLINE_VSYSCALL (getcpu, 3, cpu, node, NULL);
@@ -29,5 +30,32 @@ __getcpu (unsigned int *cpu, unsigned int *node)
return INLINE_SYSCALL_CALL (getcpu, cpu, node, NULL);
#endif
}
+
+#if defined (RSEQ_SIG) && defined (RSEQ_HAS_LOAD32_LOAD32_RELAXED)
+int
+__getcpu (unsigned int *cpu, unsigned int *node)
+{
+ /* Check if rseq is registered and the node_id feature is available. */
+ if (__glibc_likely (rseq_node_id_available()))
+ {
+ struct rseq_area *rseq_area = rseq_get_area();
+
+ if (rseq_load32_load32_relaxed(cpu, &rseq_area->cpu_id,
+ node, &rseq_area->node_id) == 0)
+ {
+ /* The critical section was not aborted, return 0. */
+ return 0;
+ }
+ }
+
+ return vsyscall_getcpu (cpu, node);
+}
+#else
+int
+__getcpu (unsigned int *cpu, unsigned int *node)
+{
+ return vsyscall_getcpu (cpu, node);
+}
+#endif
weak_alias (__getcpu, getcpu)
libc_hidden_def (__getcpu)
--
2.34.1
next prev parent reply other threads:[~2024-02-06 16:28 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-06 16:27 [PATCH v8 0/8] Extend rseq support Michael Jeanson
2024-02-06 16:27 ` [PATCH v8 1/8] nptl: fix potential merge of __rseq_* relro symbols Michael Jeanson
2024-02-06 16:41 ` Mathieu Desnoyers
2024-02-16 2:02 ` DJ Delorie
2024-02-16 20:18 ` Michael Jeanson
2024-02-16 20:29 ` DJ Delorie
2024-02-06 16:27 ` [PATCH v8 2/8] Add rseq extensible ABI support Michael Jeanson
2024-02-16 4:42 ` DJ Delorie
2024-02-16 20:19 ` Michael Jeanson
2024-02-16 20:37 ` DJ Delorie
2024-02-16 21:10 ` Michael Jeanson
2024-02-16 21:17 ` DJ Delorie
2024-02-16 21:24 ` Michael Jeanson
2024-02-06 16:27 ` [PATCH v8 3/8] nptl: Add public __rseq_feature_size symbol Michael Jeanson
2024-02-16 22:07 ` DJ Delorie
2024-02-19 19:25 ` Michael Jeanson
2024-02-19 19:43 ` DJ Delorie
2024-02-19 20:15 ` Michael Jeanson
2024-02-19 20:24 ` DJ Delorie
2024-02-19 22:06 ` Michael Jeanson
2024-02-19 22:08 ` DJ Delorie
2024-02-19 22:18 ` Michael Jeanson
2024-02-19 22:21 ` DJ Delorie
2024-02-06 16:27 ` [PATCH v8 4/8] nptl: Add features to internal 'struct rseq_area' Michael Jeanson
2024-02-17 2:04 ` DJ Delorie
2024-02-06 16:27 ` [PATCH v8 5/8] nptl: Add rseq internal utils Michael Jeanson
2024-02-17 2:33 ` DJ Delorie
2024-02-19 20:25 ` Michael Jeanson
2024-02-06 16:27 ` [PATCH v8 6/8] x86-64: Add rseq_load32_load32_relaxed Michael Jeanson
2024-02-17 3:08 ` DJ Delorie
2024-02-19 20:27 ` Michael Jeanson
2024-02-06 16:28 ` [PATCH v8 7/8] aarch64: " Michael Jeanson
2024-02-17 3:53 ` DJ Delorie
2024-02-19 20:29 ` Michael Jeanson
2024-02-20 15:07 ` Mathieu Desnoyers
2024-02-20 17:55 ` DJ Delorie
2024-02-06 16:28 ` Michael Jeanson [this message]
2024-02-17 3:57 ` [PATCH v8 8/8] Linux: Use rseq to accelerate getcpu DJ Delorie
2024-02-19 22:14 ` Michael Jeanson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240206162801.882585-9-mjeanson@efficios.com \
--to=mjeanson@efficios.com \
--cc=libc-alpha@sourceware.org \
--cc=mathieu.desnoyers@efficios.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).