* [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc
@ 2023-12-11 17:03 Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 1/6] libgomp: basic pinned memory on Linux Andrew Stubbs
` (5 more replies)
0 siblings, 6 replies; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-11 17:03 UTC (permalink / raw)
To: gcc-patches
This patch series is a rework of the v2 series I posted in August:
https://patchwork.sourceware.org/project/gcc/list/?series=23763&state=%2A&archive=both
This version addresses most of the review comments from Tobias, but
after discussion with Tobias and Thomas we've decided to skip the
nice-to-have proposed initialization improvement in the interest of
getting the job done, for now.
Otherwise, some bugs have been fixed and few other clean-ups have been
made, but the series retains the same purpose and structure.
This series no longer has any out-of-tree dependencies, now that the
low-latency allocator patch have been committed.
An older, less compact, version of these patches is already applied to
the devel/omp/gcc-13 (OG13) branch.
OK for mainline?
Andrew
Andrew Stubbs (5):
libgomp: basic pinned memory on Linux
libgomp, openmp: Add ompx_pinned_mem_alloc
openmp: Add -foffload-memory
openmp: -foffload-memory=pinned
libgomp: fine-grained pinned memory allocator
Thomas Schwinge (1):
libgomp, nvptx: Cuda pinned memory
gcc/common.opt | 16 +
gcc/coretypes.h | 7 +
gcc/doc/invoke.texi | 16 +-
gcc/omp-builtins.def | 3 +
gcc/omp-low.cc | 66 ++++
libgomp/Makefile.am | 2 +-
libgomp/Makefile.in | 7 +-
libgomp/allocator.c | 95 ++++--
libgomp/config/gcn/allocator.c | 21 +-
libgomp/config/linux/allocator.c | 243 +++++++++++++
libgomp/config/nvptx/allocator.c | 21 +-
libgomp/libgomp-plugin.h | 2 +
libgomp/libgomp.h | 14 +
libgomp/libgomp.map | 1 +
libgomp/libgomp.texi | 17 +-
libgomp/libgomp_g.h | 1 +
libgomp/omp.h.in | 1 +
libgomp/omp_lib.f90.in | 2 +
libgomp/plugin/plugin-nvptx.c | 42 +++
libgomp/target.c | 136 ++++++++
.../libgomp.c-c++-common/alloc-pinned-1.c | 28 ++
libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 141 ++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 146 ++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-3.c | 189 +++++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-4.c | 184 ++++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 129 +++++++
libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 128 +++++++
libgomp/testsuite/libgomp.c/alloc-pinned-7.c | 63 ++++
libgomp/testsuite/libgomp.c/alloc-pinned-8.c | 127 +++++++
.../libgomp.fortran/alloc-pinned-1.f90 | 16 +
libgomp/usmpin-allocator.c | 319 ++++++++++++++++++
31 files changed, 2127 insertions(+), 56 deletions(-)
create mode 100644 libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-1.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-2.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-3.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-4.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-5.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-6.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-7.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-8.c
create mode 100644 libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90
create mode 100644 libgomp/usmpin-allocator.c
--
2.41.0
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v3 1/6] libgomp: basic pinned memory on Linux
2023-12-11 17:03 [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc Andrew Stubbs
@ 2023-12-11 17:04 ` Andrew Stubbs
2023-12-12 9:02 ` Tobias Burnus
2023-12-11 17:04 ` [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc Andrew Stubbs
` (4 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-11 17:04 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 2294 bytes --]
Implement the OpenMP pinned memory trait on Linux hosts using the mlock
syscall. Pinned allocations are performed using mmap, not malloc, to ensure
that they can be unpinned safely when freed.
This implementation will work OK for page-scale allocations, and finer-grained
allocations will be implemented in a future patch.
libgomp/ChangeLog:
* allocator.c (MEMSPACE_ALLOC): Add PIN.
(MEMSPACE_CALLOC): Add PIN.
(MEMSPACE_REALLOC): Add PIN.
(MEMSPACE_FREE): Add PIN.
(MEMSPACE_VALIDATE): Add PIN.
(omp_init_allocator): Use MEMSPACE_VALIDATE to check pinning.
(omp_aligned_alloc): Add pinning to all MEMSPACE_* calls.
(omp_aligned_calloc): Likewise.
(omp_realloc): Likewise.
(omp_free): Likewise.
* config/linux/allocator.c: New file.
* config/nvptx/allocator.c (MEMSPACE_ALLOC): Add PIN.
(MEMSPACE_CALLOC): Add PIN.
(MEMSPACE_REALLOC): Add PIN.
(MEMSPACE_FREE): Add PIN.
(MEMSPACE_VALIDATE): Add PIN.
* config/gcn/allocator.c (MEMSPACE_ALLOC): Add PIN.
(MEMSPACE_CALLOC): Add PIN.
(MEMSPACE_REALLOC): Add PIN.
(MEMSPACE_FREE): Add PIN.
* libgomp.texi: Switch pinned trait to supported.
(MEMSPACE_VALIDATE): Add PIN.
* testsuite/libgomp.c/alloc-pinned-1.c: New test.
* testsuite/libgomp.c/alloc-pinned-2.c: New test.
* testsuite/libgomp.c/alloc-pinned-3.c: New test.
* testsuite/libgomp.c/alloc-pinned-4.c: New test.
Co-Authored-By: Thomas Schwinge <thomas@codesourcery.com>
---
libgomp/allocator.c | 65 +++++---
libgomp/config/gcn/allocator.c | 21 +--
libgomp/config/linux/allocator.c | 111 +++++++++++++
libgomp/config/nvptx/allocator.c | 21 +--
libgomp/libgomp.texi | 3 +-
libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 115 ++++++++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 120 ++++++++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-3.c | 156 +++++++++++++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-4.c | 150 ++++++++++++++++++
9 files changed, 716 insertions(+), 46 deletions(-)
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-1.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-2.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-3.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-4.c
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0001-libgomp-basic-pinned-memory-on-Linux.patch --]
[-- Type: text/x-patch; name="v3-0001-libgomp-basic-pinned-memory-on-Linux.patch", Size: 25450 bytes --]
diff --git a/libgomp/allocator.c b/libgomp/allocator.c
index a8a80f8028d..666adf9a3a9 100644
--- a/libgomp/allocator.c
+++ b/libgomp/allocator.c
@@ -38,27 +38,30 @@
#define omp_max_predefined_alloc omp_thread_mem_alloc
/* These macros may be overridden in config/<target>/allocator.c.
+ The defaults (no override) are to return NULL for pinned memory requests
+ and pass through to the regular OS calls otherwise.
The following definitions (ab)use comma operators to avoid unused
variable errors. */
#ifndef MEMSPACE_ALLOC
-#define MEMSPACE_ALLOC(MEMSPACE, SIZE) \
- malloc (((void)(MEMSPACE), (SIZE)))
+#define MEMSPACE_ALLOC(MEMSPACE, SIZE, PIN) \
+ (PIN ? NULL : malloc (((void)(MEMSPACE), (SIZE))))
#endif
#ifndef MEMSPACE_CALLOC
-#define MEMSPACE_CALLOC(MEMSPACE, SIZE) \
- calloc (1, (((void)(MEMSPACE), (SIZE))))
+#define MEMSPACE_CALLOC(MEMSPACE, SIZE, PIN) \
+ (PIN ? NULL : calloc (1, (((void)(MEMSPACE), (SIZE)))))
#endif
#ifndef MEMSPACE_REALLOC
-#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE) \
- realloc (ADDR, (((void)(MEMSPACE), (void)(OLDSIZE), (SIZE))))
+#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN) \
+ ((PIN) || (OLDPIN) ? NULL \
+ : realloc (ADDR, (((void)(MEMSPACE), (void)(OLDSIZE), (SIZE)))))
#endif
#ifndef MEMSPACE_FREE
-#define MEMSPACE_FREE(MEMSPACE, ADDR, SIZE) \
- free (((void)(MEMSPACE), (void)(SIZE), (ADDR)))
+#define MEMSPACE_FREE(MEMSPACE, ADDR, SIZE, PIN) \
+ if (PIN) free (((void)(MEMSPACE), (void)(SIZE), (ADDR)))
#endif
#ifndef MEMSPACE_VALIDATE
-#define MEMSPACE_VALIDATE(MEMSPACE, ACCESS) \
- (((void)(MEMSPACE), (void)(ACCESS), 1))
+#define MEMSPACE_VALIDATE(MEMSPACE, ACCESS, PIN) \
+ (PIN ? 0 : ((void)(MEMSPACE), (void)(ACCESS), 1))
#endif
/* Map the predefined allocators to the correct memory space.
@@ -439,12 +442,8 @@ omp_init_allocator (omp_memspace_handle_t memspace, int ntraits,
}
#endif
- /* No support for this so far. */
- if (data.pinned)
- return omp_null_allocator;
-
/* Reject unsupported memory spaces. */
- if (!MEMSPACE_VALIDATE (data.memspace, data.access))
+ if (!MEMSPACE_VALIDATE (data.memspace, data.access, data.pinned))
return omp_null_allocator;
ret = gomp_malloc (sizeof (struct omp_allocator_data));
@@ -586,7 +585,8 @@ retry:
}
else
#endif
- ptr = MEMSPACE_ALLOC (allocator_data->memspace, new_size);
+ ptr = MEMSPACE_ALLOC (allocator_data->memspace, new_size,
+ allocator_data->pinned);
if (ptr == NULL)
{
#ifdef HAVE_SYNC_BUILTINS
@@ -623,7 +623,8 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
- ptr = MEMSPACE_ALLOC (memspace, new_size);
+ ptr = MEMSPACE_ALLOC (memspace, new_size,
+ allocator_data && allocator_data->pinned);
}
if (ptr == NULL)
goto fail;
@@ -694,6 +695,7 @@ omp_free (void *ptr, omp_allocator_handle_t allocator)
{
struct omp_mem_header *data;
omp_memspace_handle_t memspace = omp_default_mem_space;
+ int pinned = false;
if (ptr == NULL)
return;
@@ -735,6 +737,7 @@ omp_free (void *ptr, omp_allocator_handle_t allocator)
#endif
memspace = allocator_data->memspace;
+ pinned = allocator_data->pinned;
}
else
{
@@ -759,7 +762,7 @@ omp_free (void *ptr, omp_allocator_handle_t allocator)
memspace = predefined_alloc_mapping[data->allocator];
}
- MEMSPACE_FREE (memspace, data->ptr, data->size);
+ MEMSPACE_FREE (memspace, data->ptr, data->size, pinned);
}
ialias (omp_free)
@@ -890,7 +893,8 @@ retry:
}
else
#endif
- ptr = MEMSPACE_CALLOC (allocator_data->memspace, new_size);
+ ptr = MEMSPACE_CALLOC (allocator_data->memspace, new_size,
+ allocator_data->pinned);
if (ptr == NULL)
{
#ifdef HAVE_SYNC_BUILTINS
@@ -929,7 +933,8 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
- ptr = MEMSPACE_CALLOC (memspace, new_size);
+ ptr = MEMSPACE_CALLOC (memspace, new_size,
+ allocator_data && allocator_data->pinned);
}
if (ptr == NULL)
goto fail;
@@ -1161,9 +1166,13 @@ retry:
#endif
if (prev_size)
new_ptr = MEMSPACE_REALLOC (allocator_data->memspace, data->ptr,
- data->size, new_size);
+ data->size, new_size,
+ (free_allocator_data
+ && free_allocator_data->pinned),
+ allocator_data->pinned);
else
- new_ptr = MEMSPACE_ALLOC (allocator_data->memspace, new_size);
+ new_ptr = MEMSPACE_ALLOC (allocator_data->memspace, new_size,
+ allocator_data->pinned);
if (new_ptr == NULL)
{
#ifdef HAVE_SYNC_BUILTINS
@@ -1216,10 +1225,14 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
- new_ptr = MEMSPACE_REALLOC (memspace, data->ptr, data->size, new_size);
+ new_ptr = MEMSPACE_REALLOC (memspace, data->ptr, data->size, new_size,
+ (free_allocator_data
+ && free_allocator_data->pinned),
+ allocator_data && allocator_data->pinned);
}
if (new_ptr == NULL)
goto fail;
+
ret = (char *) new_ptr + sizeof (struct omp_mem_header);
((struct omp_mem_header *) ret)[-1].ptr = new_ptr;
((struct omp_mem_header *) ret)[-1].size = new_size;
@@ -1249,7 +1262,8 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
- new_ptr = MEMSPACE_ALLOC (memspace, new_size);
+ new_ptr = MEMSPACE_ALLOC (memspace, new_size,
+ allocator_data && allocator_data->pinned);
}
if (new_ptr == NULL)
goto fail;
@@ -1304,7 +1318,8 @@ retry:
was_memspace = (free_allocator_data
? free_allocator_data->memspace
: predefined_alloc_mapping[free_allocator]);
- MEMSPACE_FREE (was_memspace, data->ptr, data->size);
+ int was_pinned = (free_allocator_data && free_allocator_data->pinned);
+ MEMSPACE_FREE (was_memspace, data->ptr, data->size, was_pinned);
}
return ret;
diff --git a/libgomp/config/gcn/allocator.c b/libgomp/config/gcn/allocator.c
index e9a95d683f9..679218f08d2 100644
--- a/libgomp/config/gcn/allocator.c
+++ b/libgomp/config/gcn/allocator.c
@@ -109,16 +109,17 @@ gcn_memspace_validate (omp_memspace_handle_t memspace, unsigned access)
|| access != omp_atv_all);
}
-#define MEMSPACE_ALLOC(MEMSPACE, SIZE) \
- gcn_memspace_alloc (MEMSPACE, SIZE)
-#define MEMSPACE_CALLOC(MEMSPACE, SIZE) \
- gcn_memspace_calloc (MEMSPACE, SIZE)
-#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE) \
- gcn_memspace_realloc (MEMSPACE, ADDR, OLDSIZE, SIZE)
-#define MEMSPACE_FREE(MEMSPACE, ADDR, SIZE) \
- gcn_memspace_free (MEMSPACE, ADDR, SIZE)
-#define MEMSPACE_VALIDATE(MEMSPACE, ACCESS) \
- gcn_memspace_validate (MEMSPACE, ACCESS)
+#define MEMSPACE_ALLOC(MEMSPACE, SIZE, PIN) \
+ gcn_memspace_alloc (MEMSPACE, ((void)(PIN), (SIZE)))
+#define MEMSPACE_CALLOC(MEMSPACE, SIZE, PIN) \
+ gcn_memspace_calloc (MEMSPACE, ((void)(PIN), (SIZE)))
+#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN) \
+ gcn_memspace_realloc (MEMSPACE, ADDR, OLDSIZE, \
+ ((void)(PIN), (void)(OLDPIN), (SIZE)))
+#define MEMSPACE_FREE(MEMSPACE, ADDR, SIZE, PIN) \
+ gcn_memspace_free (MEMSPACE, ADDR, ((void)(PIN), (SIZE)))
+#define MEMSPACE_VALIDATE(MEMSPACE, ACCESS, PIN) \
+ gcn_memspace_validate (MEMSPACE, ((void)(PIN), (ACCESS)))
/* The default low-latency memspace implies omp_atv_all, which is incompatible
with the LDS memory space. */
diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c
index 64b1b4b9623..269d0d607d8 100644
--- a/libgomp/config/linux/allocator.c
+++ b/libgomp/config/linux/allocator.c
@@ -34,4 +34,115 @@
#define LIBGOMP_USE_LIBNUMA
#endif
+/* Implement malloc routines that can handle pinned memory on Linux.
+
+ It's possible to use mlock on any heap memory, but using munlock is
+ problematic if there are multiple pinned allocations on the same page.
+ Tracking all that manually would be possible, but adds overhead. This may
+ be worth it if there are a lot of small allocations getting pinned, but
+ this seems less likely in a HPC application.
+
+ Instead we optimize for large pinned allocations, and use mmap to ensure
+ that two pinned allocations don't share the same page. This also means
+ that large allocations don't pin extra pages by being poorly aligned. */
+
+#define _GNU_SOURCE
+#include <sys/mman.h>
+#include <string.h>
+#include "libgomp.h"
+
+static void *
+linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin)
+{
+ (void)memspace;
+
+ if (pin)
+ {
+ /* Note that mmap always returns zeroed memory and is therefore also a
+ suitable implementation of calloc. */
+ void *addr = mmap (NULL, size, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (addr == MAP_FAILED)
+ return NULL;
+
+ if (mlock (addr, size))
+ {
+ gomp_debug (0, "libgomp: failed to pin %ld bytes of"
+ " memory (ulimit too low?)\n", size);
+ munmap (addr, size);
+ return NULL;
+ }
+
+ return addr;
+ }
+ else
+ return malloc (size);
+}
+
+static void *
+linux_memspace_calloc (omp_memspace_handle_t memspace, size_t size, int pin)
+{
+ if (pin)
+ return linux_memspace_alloc (memspace, size, pin);
+ else
+ return calloc (1, size);
+}
+
+static void
+linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size,
+ int pin)
+{
+ (void)memspace;
+
+ if (pin)
+ munmap (addr, size);
+ else
+ free (addr);
+}
+
+static void *
+linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr,
+ size_t oldsize, size_t size, int oldpin, int pin)
+{
+ if (oldpin && pin)
+ {
+ void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE);
+ if (newaddr == MAP_FAILED)
+ return NULL;
+
+ return newaddr;
+ }
+ else if (oldpin || pin)
+ {
+ void *newaddr = linux_memspace_alloc (memspace, size, pin);
+ if (newaddr)
+ {
+ memcpy (newaddr, addr, oldsize < size ? oldsize : size);
+ linux_memspace_free (memspace, addr, oldsize, oldpin);
+ }
+
+ return newaddr;
+ }
+ else
+ return realloc (addr, size);
+}
+
+static int
+linux_memspace_validate (omp_memspace_handle_t, unsigned, int)
+{
+ /* Everything should be accepted on Linux, including pinning. */
+ return 1;
+}
+
+#define MEMSPACE_ALLOC(MEMSPACE, SIZE, PIN) \
+ linux_memspace_alloc (MEMSPACE, SIZE, PIN)
+#define MEMSPACE_CALLOC(MEMSPACE, SIZE, PIN) \
+ linux_memspace_calloc (MEMSPACE, SIZE, PIN)
+#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN) \
+ linux_memspace_realloc (MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN)
+#define MEMSPACE_FREE(MEMSPACE, ADDR, SIZE, PIN) \
+ linux_memspace_free (MEMSPACE, ADDR, SIZE, PIN)
+#define MEMSPACE_VALIDATE(MEMSPACE, ACCESS, PIN) \
+ linux_memspace_validate (MEMSPACE, ACCESS, PIN)
+
#include "../../allocator.c"
diff --git a/libgomp/config/nvptx/allocator.c b/libgomp/config/nvptx/allocator.c
index a3302411bcb..6a226a81b75 100644
--- a/libgomp/config/nvptx/allocator.c
+++ b/libgomp/config/nvptx/allocator.c
@@ -123,16 +123,17 @@ nvptx_memspace_validate (omp_memspace_handle_t memspace, unsigned access)
#endif
}
-#define MEMSPACE_ALLOC(MEMSPACE, SIZE) \
- nvptx_memspace_alloc (MEMSPACE, SIZE)
-#define MEMSPACE_CALLOC(MEMSPACE, SIZE) \
- nvptx_memspace_calloc (MEMSPACE, SIZE)
-#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE) \
- nvptx_memspace_realloc (MEMSPACE, ADDR, OLDSIZE, SIZE)
-#define MEMSPACE_FREE(MEMSPACE, ADDR, SIZE) \
- nvptx_memspace_free (MEMSPACE, ADDR, SIZE)
-#define MEMSPACE_VALIDATE(MEMSPACE, ACCESS) \
- nvptx_memspace_validate (MEMSPACE, ACCESS)
+#define MEMSPACE_ALLOC(MEMSPACE, SIZE, PIN) \
+ nvptx_memspace_alloc (MEMSPACE, ((void)(PIN), (SIZE)))
+#define MEMSPACE_CALLOC(MEMSPACE, SIZE, PIN) \
+ nvptx_memspace_calloc (MEMSPACE, ((void)(PIN), (SIZE)))
+#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN) \
+ nvptx_memspace_realloc (MEMSPACE, ADDR, OLDSIZE, \
+ ((void)(OLDPIN), (void)(PIN), (SIZE)))
+#define MEMSPACE_FREE(MEMSPACE, ADDR, SIZE, PIN) \
+ nvptx_memspace_free (MEMSPACE, ADDR, ((void)(PIN), (SIZE)))
+#define MEMSPACE_VALIDATE(MEMSPACE, ACCESS, PIN) \
+ nvptx_memspace_validate (MEMSPACE, ((void)(PIN), (ACCESS)))
/* The default low-latency memspace implies omp_atv_all, which is incompatible
with the .shared memory space. */
diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi
index 67a111265a0..5838311b7d8 100644
--- a/libgomp/libgomp.texi
+++ b/libgomp/libgomp.texi
@@ -5751,7 +5751,8 @@ a @code{nearest} allocation.
Additional notes regarding the traits:
@itemize
-@item The @code{pinned} trait is unsupported.
+@item The @code{pinned} trait is supported on Linux hosts, but is subject to
+ the OS @code{ulimit}/@code{rlimit} locked memory settings.
@item The default for the @code{pool_size} trait is no pool and for every
(re)allocation the associated library routine is called, which might
internally use a memory pool.
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c
new file mode 100644
index 00000000000..e17a21f0a6c
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c
@@ -0,0 +1,115 @@
+/* { dg-do run } */
+
+/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+
+/* Test that pinned memory works. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+#include <sys/resource.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGESIZE)
+#define CHECK_SIZE(SIZE) { \
+ struct rlimit limit; \
+ if (getrlimit (RLIMIT_MEMLOCK, &limit) \
+ || limit.rlim_cur <= SIZE) \
+ fprintf (stderr, "unsufficient lockable memory; please increase ulimit\n"); \
+ }
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+#else
+#define PAGE_SIZE 1024 /* unknown */
+#define CHECK_SIZE(SIZE) fprintf (stderr, "OS unsupported\n");
+#define EXPECT_OMP_NULL_ALLOCATOR
+
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+#endif
+
+static void
+verify0 (char *p, size_t s)
+{
+ for (size_t i = 0; i < s; ++i)
+ if (p[i] != 0)
+ abort ();
+}
+
+#include <omp.h>
+
+int
+main ()
+{
+ /* Allocate at least a page each time, allowing space for overhead,
+ but stay within the ulimit. */
+ const int SIZE = PAGE_SIZE - 128;
+ CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures
+
+ const omp_alloctrait_t traits[] = {
+ { omp_atk_pinned, 1 }
+ };
+ omp_allocator_handle_t allocator = omp_init_allocator (omp_default_mem_space,
+ 1, traits);
+
+#ifdef EXPECT_OMP_NULL_ALLOCATOR
+ if (allocator == omp_null_allocator)
+ return 0;
+#endif
+
+ // Sanity check
+ if (get_pinned_mem () != 0)
+ abort ();
+
+ void *p = omp_alloc (SIZE, allocator);
+ if (!p)
+ abort ();
+
+ int amount = get_pinned_mem ();
+ if (amount == 0)
+ abort ();
+
+ p = omp_realloc (p, SIZE * 2, allocator, allocator);
+
+ int amount2 = get_pinned_mem ();
+ if (amount2 <= amount)
+ abort ();
+
+ /* SIZE*2 ensures that it doesn't slot into the space possibly
+ vacated by realloc. */
+ p = omp_calloc (1, SIZE * 2, allocator);
+
+ if (get_pinned_mem () <= amount2)
+ abort ();
+
+ verify0 (p, SIZE * 2);
+
+ return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c
new file mode 100644
index 00000000000..3cf322cfbc8
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c
@@ -0,0 +1,120 @@
+/* { dg-do run } */
+
+/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+
+/* Test that pinned memory works (pool_size code path). */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+#include <sys/resource.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGESIZE)
+#define CHECK_SIZE(SIZE) { \
+ struct rlimit limit; \
+ if (getrlimit (RLIMIT_MEMLOCK, &limit) \
+ || limit.rlim_cur <= SIZE) \
+ fprintf (stderr, "unsufficient lockable memory; please increase ulimit\n"); \
+ }
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+#else
+#define PAGE_SIZE 1024 /* unknown */
+#define CHECK_SIZE(SIZE) fprintf (stderr, "OS unsupported\n");
+#define EXPECT_OMP_NULL_ALLOCATOR
+
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+#endif
+
+static void
+verify0 (char *p, size_t s)
+{
+ for (size_t i = 0; i < s; ++i)
+ if (p[i] != 0)
+ abort ();
+}
+
+#include <omp.h>
+
+int
+main ()
+{
+ /* Allocate at least a page each time, allowing space for overhead,
+ but stay within the ulimit. */
+ const int SIZE = PAGE_SIZE - 128;
+ CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures
+
+ const omp_alloctrait_t traits[] = {
+ { omp_atk_pinned, 1 },
+ { omp_atk_pool_size, SIZE * 8 }
+ };
+ omp_allocator_handle_t allocator = omp_init_allocator (omp_default_mem_space,
+ 2, traits);
+
+#ifdef EXPECT_OMP_NULL_ALLOCATOR
+ if (allocator == omp_null_allocator)
+ return 0;
+#endif
+
+ // Sanity check
+ if (get_pinned_mem () != 0)
+ abort ();
+
+ void *p = omp_alloc (SIZE, allocator);
+ if (!p)
+ abort ();
+
+ int amount = get_pinned_mem ();
+ if (amount == 0)
+ abort ();
+
+ p = omp_realloc (p, SIZE * 2, allocator, allocator);
+ if (!p)
+ abort ();
+
+ int amount2 = get_pinned_mem ();
+ if (amount2 <= amount)
+ abort ();
+
+ /* SIZE*2 ensures that it doesn't slot into the space possibly
+ vacated by realloc. */
+ p = omp_calloc (1, SIZE * 2, allocator);
+ if (!p)
+ abort ();
+
+ if (get_pinned_mem () <= amount2)
+ abort ();
+
+ verify0 (p, SIZE * 2);
+
+ return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-3.c b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c
new file mode 100644
index 00000000000..53e4720cc9c
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c
@@ -0,0 +1,156 @@
+/* { dg-do run } */
+
+/* Test that pinned memory fails correctly. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+#include <sys/resource.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGESIZE)
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+
+void
+set_pin_limit (int size)
+{
+ struct rlimit limit;
+ if (getrlimit (RLIMIT_MEMLOCK, &limit))
+ abort ();
+ limit.rlim_cur = (limit.rlim_max < size ? limit.rlim_max : size);
+ if (setrlimit (RLIMIT_MEMLOCK, &limit))
+ abort ();
+}
+#else
+#define PAGE_SIZE 10000 * 1024 /* unknown */
+#define EXPECT_OMP_NULL_ALLOCATOR
+
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+
+void
+set_pin_limit ()
+{
+}
+#endif
+
+static void
+verify0 (char *p, size_t s)
+{
+ for (size_t i = 0; i < s; ++i)
+ if (p[i] != 0)
+ abort ();
+}
+
+#include <omp.h>
+
+int
+main ()
+{
+ /* This needs to be large enough to cover multiple pages. */
+ const int SIZE = PAGE_SIZE * 4;
+
+ /* Pinned memory, no fallback. */
+ const omp_alloctrait_t traits1[] = {
+ { omp_atk_pinned, 1 },
+ { omp_atk_fallback, omp_atv_null_fb }
+ };
+ omp_allocator_handle_t allocator1 = omp_init_allocator (omp_default_mem_space,
+ 2, traits1);
+
+ /* Pinned memory, plain memory fallback. */
+ const omp_alloctrait_t traits2[] = {
+ { omp_atk_pinned, 1 },
+ { omp_atk_fallback, omp_atv_default_mem_fb }
+ };
+ omp_allocator_handle_t allocator2 = omp_init_allocator (omp_default_mem_space,
+ 2, traits2);
+
+#ifdef EXPECT_OMP_NULL_ALLOCATOR
+ if (allocator1 == omp_null_allocator
+ && allocator2 == omp_null_allocator)
+ return 0;
+#endif
+
+ /* Ensure that the limit is smaller than the allocation. */
+ set_pin_limit (SIZE / 2);
+
+ // Sanity check
+ if (get_pinned_mem () != 0)
+ abort ();
+
+ // Should fail
+ void *p1 = omp_alloc (SIZE, allocator1);
+ if (p1)
+ abort ();
+
+ // Should fail
+ void *p2 = omp_calloc (1, SIZE, allocator1);
+ if (p2)
+ abort ();
+
+ // Should fall back
+ void *p3 = omp_alloc (SIZE, allocator2);
+ if (!p3)
+ abort ();
+
+ // Should fall back
+ void *p4 = omp_calloc (1, SIZE, allocator2);
+ if (!p4)
+ abort ();
+ verify0 (p4, SIZE);
+
+ // Should fail to realloc
+ void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc);
+ void *p5 = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc);
+ if (!notpinned || p5)
+ abort ();
+
+ // Should fall back to no realloc needed
+ void *p6 = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc);
+ if (p6 != notpinned)
+ abort ();
+
+ // No memory should have been pinned
+ int amount = get_pinned_mem ();
+ if (amount != 0)
+ abort ();
+
+ // Ensure free works correctly
+ if (p1) omp_free (p1, allocator1);
+ if (p2) omp_free (p2, allocator1);
+ if (p3) omp_free (p3, allocator2);
+ if (p4) omp_free (p4, allocator2);
+ // p5 and notpinned have been reallocated
+ if (p6) omp_free (p6, omp_default_mem_alloc);
+
+ return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-4.c b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c
new file mode 100644
index 00000000000..9d850c23e4b
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c
@@ -0,0 +1,150 @@
+/* { dg-do run } */
+
+/* Test that pinned memory fails correctly, pool_size code path. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+#include <sys/resource.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGESIZE)
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+
+void
+set_pin_limit (int size)
+{
+ struct rlimit limit;
+ if (getrlimit (RLIMIT_MEMLOCK, &limit))
+ abort ();
+ limit.rlim_cur = (limit.rlim_max < size ? limit.rlim_max : size);
+ if (setrlimit (RLIMIT_MEMLOCK, &limit))
+ abort ();
+}
+#else
+#define PAGE_SIZE 10000 * 1024 /* unknown */
+#define EXPECT_OMP_NULL_ALLOCATOR
+
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+
+void
+set_pin_limit ()
+{
+}
+#endif
+
+static void
+verify0 (char *p, size_t s)
+{
+ for (size_t i = 0; i < s; ++i)
+ if (p[i] != 0)
+ abort ();
+}
+
+#include <omp.h>
+
+int
+main ()
+{
+ /* This needs to be large enough to cover multiple pages. */
+ const int SIZE = PAGE_SIZE * 4;
+
+ /* Pinned memory, no fallback. */
+ const omp_alloctrait_t traits1[] = {
+ { omp_atk_pinned, 1 },
+ { omp_atk_fallback, omp_atv_null_fb },
+ { omp_atk_pool_size, SIZE * 8 }
+ };
+ omp_allocator_handle_t allocator1 = omp_init_allocator (omp_default_mem_space,
+ 3, traits1);
+
+ /* Pinned memory, plain memory fallback. */
+ const omp_alloctrait_t traits2[] = {
+ { omp_atk_pinned, 1 },
+ { omp_atk_fallback, omp_atv_default_mem_fb },
+ { omp_atk_pool_size, SIZE * 8 }
+ };
+ omp_allocator_handle_t allocator2 = omp_init_allocator (omp_default_mem_space,
+ 3, traits2);
+
+#ifdef EXPECT_OMP_NULL_ALLOCATOR
+ if (allocator1 == omp_null_allocator
+ && allocator2 == omp_null_allocator)
+ return 0;
+#endif
+
+ /* Ensure that the limit is smaller than the allocation. */
+ set_pin_limit (SIZE / 2);
+
+ // Sanity check
+ if (get_pinned_mem () != 0)
+ abort ();
+
+ // Should fail
+ void *p = omp_alloc (SIZE, allocator1);
+ if (p)
+ abort ();
+
+ // Should fail
+ p = omp_calloc (1, SIZE, allocator1);
+ if (p)
+ abort ();
+
+ // Should fall back
+ p = omp_alloc (SIZE, allocator2);
+ if (!p)
+ abort ();
+
+ // Should fall back
+ p = omp_calloc (1, SIZE, allocator2);
+ if (!p)
+ abort ();
+ verify0 (p, SIZE);
+
+ // Should fail to realloc
+ void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc);
+ p = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc);
+ if (!notpinned || p)
+ abort ();
+
+ // Should fall back to no realloc needed
+ p = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc);
+ if (p != notpinned)
+ abort ();
+
+ // No memory should have been pinned
+ int amount = get_pinned_mem ();
+ if (amount != 0)
+ abort ();
+
+ return 0;
+}
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc
2023-12-11 17:03 [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 1/6] libgomp: basic pinned memory on Linux Andrew Stubbs
@ 2023-12-11 17:04 ` Andrew Stubbs
2023-12-12 10:05 ` Tobias Burnus
2023-12-11 17:04 ` [PATCH v3 3/6] openmp: Add -foffload-memory Andrew Stubbs
` (3 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-11 17:04 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]
This creates a new predefined allocator as a shortcut for using pinned
memory with OpenMP. The name uses the OpenMP extension space and is
intended to be consistent with other OpenMP implementations currently in
development.
The allocator is equivalent to using a custom allocator with the pinned
trait and the null fallback trait.
libgomp/ChangeLog:
* allocator.c (omp_max_predefined_alloc): Update.
(predefined_alloc_mapping): Add ompx_pinned_mem_alloc entry.
(omp_aligned_alloc): Support ompx_pinned_mem_alloc.
(omp_free): Likewise.
(omp_aligned_calloc): Likewise.
(omp_realloc): Likewise.
* libgomp.texi: Document ompx_pinned_mem_alloc.
* omp.h.in (omp_allocator_handle_t): Add ompx_pinned_mem_alloc.
* omp_lib.f90.in: Add ompx_pinned_mem_alloc.
* testsuite/libgomp.c/alloc-pinned-5.c: New test.
* testsuite/libgomp.c/alloc-pinned-6.c: New test.
* testsuite/libgomp.fortran/alloc-pinned-1.f90: New test.
Co-Authored-By: Thomas Schwinge <thomas@codesourcery.com>
---
libgomp/allocator.c | 58 ++++++----
libgomp/libgomp.texi | 7 +-
libgomp/omp.h.in | 1 +
libgomp/omp_lib.f90.in | 2 +
libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 103 ++++++++++++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 101 +++++++++++++++++
.../libgomp.fortran/alloc-pinned-1.f90 | 16 +++
7 files changed, 268 insertions(+), 20 deletions(-)
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-5.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-6.c
create mode 100644 libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0002-libgomp-openmp-Add-ompx_pinned_mem_alloc.patch --]
[-- Type: text/x-patch; name="v3-0002-libgomp-openmp-Add-ompx_pinned_mem_alloc.patch", Size: 12974 bytes --]
diff --git a/libgomp/allocator.c b/libgomp/allocator.c
index 666adf9a3a9..6c69c4f008f 100644
--- a/libgomp/allocator.c
+++ b/libgomp/allocator.c
@@ -35,7 +35,7 @@
#include <dlfcn.h>
#endif
-#define omp_max_predefined_alloc omp_thread_mem_alloc
+#define omp_max_predefined_alloc ompx_pinned_mem_alloc
/* These macros may be overridden in config/<target>/allocator.c.
The defaults (no override) are to return NULL for pinned memory requests
@@ -78,6 +78,7 @@ static const omp_memspace_handle_t predefined_alloc_mapping[] = {
omp_low_lat_mem_space, /* omp_cgroup_mem_alloc (implementation defined). */
omp_low_lat_mem_space, /* omp_pteam_mem_alloc (implementation defined). */
omp_low_lat_mem_space, /* omp_thread_mem_alloc (implementation defined). */
+ omp_default_mem_space, /* ompx_pinned_mem_alloc. */
};
#define ARRAY_SIZE(A) (sizeof (A) / sizeof ((A)[0]))
@@ -623,8 +624,10 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
- ptr = MEMSPACE_ALLOC (memspace, new_size,
- allocator_data && allocator_data->pinned);
+ int pinned = (allocator_data
+ ? allocator_data->pinned
+ : allocator == ompx_pinned_mem_alloc);
+ ptr = MEMSPACE_ALLOC (memspace, new_size, pinned);
}
if (ptr == NULL)
goto fail;
@@ -645,7 +648,8 @@ retry:
fail:;
int fallback = (allocator_data
? allocator_data->fallback
- : allocator == omp_default_mem_alloc
+ : (allocator == omp_default_mem_alloc
+ || allocator == ompx_pinned_mem_alloc)
? omp_atv_null_fb
: omp_atv_default_mem_fb);
switch (fallback)
@@ -760,6 +764,7 @@ omp_free (void *ptr, omp_allocator_handle_t allocator)
#endif
memspace = predefined_alloc_mapping[data->allocator];
+ pinned = (data->allocator == ompx_pinned_mem_alloc);
}
MEMSPACE_FREE (memspace, data->ptr, data->size, pinned);
@@ -933,8 +938,10 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
- ptr = MEMSPACE_CALLOC (memspace, new_size,
- allocator_data && allocator_data->pinned);
+ int pinned = (allocator_data
+ ? allocator_data->pinned
+ : allocator == ompx_pinned_mem_alloc);
+ ptr = MEMSPACE_CALLOC (memspace, new_size, pinned);
}
if (ptr == NULL)
goto fail;
@@ -955,7 +962,8 @@ retry:
fail:;
int fallback = (allocator_data
? allocator_data->fallback
- : allocator == omp_default_mem_alloc
+ : (allocator == omp_default_mem_alloc
+ || allocator == ompx_pinned_mem_alloc)
? omp_atv_null_fb
: omp_atv_default_mem_fb);
switch (fallback)
@@ -1165,11 +1173,14 @@ retry:
else
#endif
if (prev_size)
- new_ptr = MEMSPACE_REALLOC (allocator_data->memspace, data->ptr,
- data->size, new_size,
- (free_allocator_data
- && free_allocator_data->pinned),
- allocator_data->pinned);
+ {
+ int was_pinned = (free_allocator_data
+ ? free_allocator_data->pinned
+ : free_allocator == ompx_pinned_mem_alloc);
+ new_ptr = MEMSPACE_REALLOC (allocator_data->memspace, data->ptr,
+ data->size, new_size, was_pinned,
+ allocator_data->pinned);
+ }
else
new_ptr = MEMSPACE_ALLOC (allocator_data->memspace, new_size,
allocator_data->pinned);
@@ -1225,10 +1236,14 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
+ int was_pinned = (free_allocator_data
+ ? free_allocator_data->pinned
+ : free_allocator == ompx_pinned_mem_alloc);
+ int pinned = (allocator_data
+ ? allocator_data->pinned
+ : allocator == ompx_pinned_mem_alloc);
new_ptr = MEMSPACE_REALLOC (memspace, data->ptr, data->size, new_size,
- (free_allocator_data
- && free_allocator_data->pinned),
- allocator_data && allocator_data->pinned);
+ was_pinned, pinned);
}
if (new_ptr == NULL)
goto fail;
@@ -1262,8 +1277,10 @@ retry:
memspace = (allocator_data
? allocator_data->memspace
: predefined_alloc_mapping[allocator]);
- new_ptr = MEMSPACE_ALLOC (memspace, new_size,
- allocator_data && allocator_data->pinned);
+ int pinned = (allocator_data
+ ? allocator_data->pinned
+ : allocator == ompx_pinned_mem_alloc);
+ new_ptr = MEMSPACE_ALLOC (memspace, new_size, pinned);
}
if (new_ptr == NULL)
goto fail;
@@ -1318,7 +1335,9 @@ retry:
was_memspace = (free_allocator_data
? free_allocator_data->memspace
: predefined_alloc_mapping[free_allocator]);
- int was_pinned = (free_allocator_data && free_allocator_data->pinned);
+ int was_pinned = (free_allocator_data
+ ? free_allocator_data->pinned
+ : free_allocator == ompx_pinned_mem_alloc);
MEMSPACE_FREE (was_memspace, data->ptr, data->size, was_pinned);
}
return ret;
@@ -1326,7 +1345,8 @@ retry:
fail:;
int fallback = (allocator_data
? allocator_data->fallback
- : allocator == omp_default_mem_alloc
+ : (allocator == omp_default_mem_alloc
+ || allocator == ompx_pinned_mem_alloc)
? omp_atv_null_fb
: omp_atv_default_mem_fb);
switch (fallback)
diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi
index 5838311b7d8..1f73c49c9b8 100644
--- a/libgomp/libgomp.texi
+++ b/libgomp/libgomp.texi
@@ -3015,6 +3015,7 @@ value.
@item omp_cgroup_mem_alloc @tab omp_low_lat_mem_space (implementation defined)
@item omp_pteam_mem_alloc @tab omp_low_lat_mem_space (implementation defined)
@item omp_thread_mem_alloc @tab omp_low_lat_mem_space (implementation defined)
+@item ompx_pinned_mem_alloc @tab omp_default_mem_space (GNU extension)
@end multitable
The predefined allocators use the default values for the traits,
@@ -3040,7 +3041,7 @@ as listed below. Except that the last three allocators have the
@item @code{fb_data} @tab @emph{unsupported as it needs an allocator handle}
@tab (none)
@item @code{pinned} @tab @code{true}, @code{false}
- @tab @code{false}
+ @tab See below
@item @code{partition} @tab @code{environment}, @code{nearest},
@code{blocked}, @code{interleaved}
@tab @code{environment}
@@ -3051,6 +3052,10 @@ For the @code{fallback} trait, the default value is @code{null_fb} for the
with device memory; for all other other allocators, it is @code{default_mem_fb}
by default.
+For the @code{pinned} trait, the default value is @code{true} for
+predefined allocator @code{ompx_pinned_mem_alloc} (a GNU extension), and
+@code{false} for all others.
+
Examples:
@smallexample
OMP_ALLOCATOR=omp_high_bw_mem_alloc
diff --git a/libgomp/omp.h.in b/libgomp/omp.h.in
index bd1286c2a3f..66989e693b7 100644
--- a/libgomp/omp.h.in
+++ b/libgomp/omp.h.in
@@ -134,6 +134,7 @@ typedef enum omp_allocator_handle_t __GOMP_UINTPTR_T_ENUM
omp_cgroup_mem_alloc = 6,
omp_pteam_mem_alloc = 7,
omp_thread_mem_alloc = 8,
+ ompx_pinned_mem_alloc = 9,
__omp_allocator_handle_t_max__ = __UINTPTR_MAX__
} omp_allocator_handle_t;
diff --git a/libgomp/omp_lib.f90.in b/libgomp/omp_lib.f90.in
index 755c0bf16ec..e27e4aacbdf 100644
--- a/libgomp/omp_lib.f90.in
+++ b/libgomp/omp_lib.f90.in
@@ -158,6 +158,8 @@
parameter :: omp_pteam_mem_alloc = 7
integer (kind=omp_allocator_handle_kind), &
parameter :: omp_thread_mem_alloc = 8
+ integer (kind=omp_allocator_handle_kind), &
+ parameter :: ompx_pinned_mem_alloc = 9
integer (omp_memspace_handle_kind), &
parameter :: omp_default_mem_space = 0
integer (omp_memspace_handle_kind), &
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c
new file mode 100644
index 00000000000..18e6d20ca5b
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c
@@ -0,0 +1,103 @@
+/* { dg-do run } */
+
+/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+
+/* Test that ompx_pinned_mem_alloc works. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+#include <sys/resource.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGESIZE)
+#define CHECK_SIZE(SIZE) { \
+ struct rlimit limit; \
+ if (getrlimit (RLIMIT_MEMLOCK, &limit) \
+ || limit.rlim_cur <= SIZE) \
+ fprintf (stderr, "unsufficient lockable memory; please increase ulimit\n"); \
+ }
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+#else
+#define PAGE_SIZE 10000 * 1024 /* unknown */
+#define CHECK_SIZE(SIZE) fprintf (stderr, "OS unsupported\n");
+
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+#endif
+
+static void
+verify0 (char *p, size_t s)
+{
+ for (size_t i = 0; i < s; ++i)
+ if (p[i] != 0)
+ abort ();
+}
+
+#include <omp.h>
+
+int
+main ()
+{
+ /* Allocate at least a page each time, allowing space for overhead,
+ but stay within the ulimit. */
+ const int SIZE = PAGE_SIZE - 128;
+ CHECK_SIZE (SIZE * 5);
+
+ // Sanity check
+ if (get_pinned_mem () != 0)
+ abort ();
+
+ void *p = omp_alloc (SIZE, ompx_pinned_mem_alloc);
+ if (!p)
+ abort ();
+
+ int amount = get_pinned_mem ();
+ if (amount == 0)
+ abort ();
+
+ p = omp_realloc (p, SIZE * 2, ompx_pinned_mem_alloc, ompx_pinned_mem_alloc);
+
+ int amount2 = get_pinned_mem ();
+ if (amount2 <= amount)
+ abort ();
+
+ /* SIZE*2 ensures that it doesn't slot into the space possibly
+ vacated by realloc. */
+ p = omp_calloc (1, SIZE * 2, ompx_pinned_mem_alloc);
+
+ if (get_pinned_mem () <= amount2)
+ abort ();
+
+ verify0 (p, SIZE * 2);
+
+ return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c
new file mode 100644
index 00000000000..f80a0264f97
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c
@@ -0,0 +1,101 @@
+/* { dg-do run } */
+
+/* Test that ompx_pinned_mem_alloc fails correctly. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+#include <sys/resource.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGESIZE)
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+
+void
+set_pin_limit (int size)
+{
+ struct rlimit limit;
+ if (getrlimit (RLIMIT_MEMLOCK, &limit))
+ abort ();
+ limit.rlim_cur = (limit.rlim_max < size ? limit.rlim_max : size);
+ if (setrlimit (RLIMIT_MEMLOCK, &limit))
+ abort ();
+}
+#else
+#define PAGE_SIZE 10000 * 1024 /* unknown */
+
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+
+void
+set_pin_limit ()
+{
+}
+#endif
+
+#include <omp.h>
+
+int
+main ()
+{
+ /* Allocate at least a page each time, but stay within the ulimit. */
+ const int SIZE = PAGE_SIZE * 4;
+
+ /* Ensure that the limit is smaller than the allocation. */
+ set_pin_limit (SIZE / 2);
+
+ // Sanity check
+ if (get_pinned_mem () != 0)
+ abort ();
+
+ // Should fail
+ void *p = omp_alloc (SIZE, ompx_pinned_mem_alloc);
+ if (p)
+ abort ();
+
+ // Should fail
+ p = omp_calloc (1, SIZE, ompx_pinned_mem_alloc);
+ if (p)
+ abort ();
+
+ // Should fail to realloc
+ void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc);
+ p = omp_realloc (notpinned, SIZE, ompx_pinned_mem_alloc, omp_default_mem_alloc);
+ if (!notpinned || p)
+ abort ();
+
+ // No memory should have been pinned
+ int amount = get_pinned_mem ();
+ if (amount != 0)
+ abort ();
+
+ return 0;
+}
diff --git a/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90 b/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90
new file mode 100644
index 00000000000..798dc3d5a12
--- /dev/null
+++ b/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90
@@ -0,0 +1,16 @@
+! Ensure that the ompx_pinned_mem_alloc predefined allocator is present and
+! accepted. The majority of the functionality testing lives in the C tests.
+!
+! { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } }
+
+program main
+ use omp_lib
+ use ISO_C_Binding
+ implicit none (external, type)
+
+ type(c_ptr) :: p
+
+ p = omp_alloc (10_c_size_t, ompx_pinned_mem_alloc);
+ if (.not. c_associated (p)) stop 1
+ call omp_free (p, ompx_pinned_mem_alloc);
+end program main
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v3 3/6] openmp: Add -foffload-memory
2023-12-11 17:03 [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 1/6] libgomp: basic pinned memory on Linux Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc Andrew Stubbs
@ 2023-12-11 17:04 ` Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 4/6] openmp: -foffload-memory=pinned Andrew Stubbs
` (2 subsequent siblings)
5 siblings, 0 replies; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-11 17:04 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 409 bytes --]
Add a new option. It's inactive until I add some follow-up patches.
gcc/ChangeLog:
* common.opt: Add -foffload-memory and its enum values.
* coretypes.h (enum offload_memory): New.
* doc/invoke.texi: Document -foffload-memory.
---
gcc/common.opt | 16 ++++++++++++++++
gcc/coretypes.h | 7 +++++++
gcc/doc/invoke.texi | 16 +++++++++++++++-
3 files changed, 38 insertions(+), 1 deletion(-)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0003-openmp-Add-foffload-memory.patch --]
[-- Type: text/x-patch; name="v3-0003-openmp-Add-foffload-memory.patch", Size: 2914 bytes --]
diff --git a/gcc/common.opt b/gcc/common.opt
index 5eb5ecff04b..a008827cfa2 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -2332,6 +2332,22 @@ Enum(offload_abi) String(ilp32) Value(OFFLOAD_ABI_ILP32)
EnumValue
Enum(offload_abi) String(lp64) Value(OFFLOAD_ABI_LP64)
+foffload-memory=
+Common Joined RejectNegative Enum(offload_memory) Var(flag_offload_memory) Init(OFFLOAD_MEMORY_NONE)
+-foffload-memory=[none|unified|pinned] Use an offload memory optimization.
+
+Enum
+Name(offload_memory) Type(enum offload_memory) UnknownError(Unknown offload memory option %qs)
+
+EnumValue
+Enum(offload_memory) String(none) Value(OFFLOAD_MEMORY_NONE)
+
+EnumValue
+Enum(offload_memory) String(unified) Value(OFFLOAD_MEMORY_UNIFIED)
+
+EnumValue
+Enum(offload_memory) String(pinned) Value(OFFLOAD_MEMORY_PINNED)
+
fomit-frame-pointer
Common Var(flag_omit_frame_pointer) Optimization
When possible do not generate stack frames.
diff --git a/gcc/coretypes.h b/gcc/coretypes.h
index fe5b868fb4f..fb4bf37ba24 100644
--- a/gcc/coretypes.h
+++ b/gcc/coretypes.h
@@ -218,6 +218,13 @@ enum offload_abi {
OFFLOAD_ABI_ILP32
};
+/* Types of memory optimization for an offload device. */
+enum offload_memory {
+ OFFLOAD_MEMORY_NONE,
+ OFFLOAD_MEMORY_UNIFIED,
+ OFFLOAD_MEMORY_PINNED
+};
+
/* Types of profile update methods. */
enum profile_update {
PROFILE_UPDATE_SINGLE,
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 43341fe6e5e..f6a7459bda7 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -202,7 +202,7 @@ in the following sections.
-fno-builtin -fno-builtin-@var{function} -fcond-mismatch
-ffreestanding -fgimple -fgnu-tm -fgnu89-inline -fhosted
-flax-vector-conversions -fms-extensions
--foffload=@var{arg} -foffload-options=@var{arg}
+-foffload=@var{arg} -foffload-options=@var{arg} -foffload-memory=@var{arg}
-fopenacc -fopenacc-dim=@var{geom}
-fopenmp -fopenmp-simd -fopenmp-target-simd-clone@r{[}=@var{device-type}@r{]}
-fpermitted-flt-eval-methods=@var{standard}
@@ -2766,6 +2766,20 @@ Typical command lines are
-foffload-options=amdgcn-amdhsa=-march=gfx906
@end smallexample
+@opindex foffload-memory
+@cindex OpenMP offloading memory modes
+@item -foffload-memory=none
+@itemx -foffload-memory=unified
+@itemx -foffload-memory=pinned
+Enable a memory optimization mode to use with OpenMP. The default behavior,
+@option{-foffload-memory=none}, is to do nothing special (unless enabled via
+a requires directive in the code). @option{-foffload-memory=unified} is
+equivalent to @code{#pragma omp requires unified_shared_memory}.
+@option{-foffload-memory=pinned} forces all host memory to be pinned (this
+mode may require the user to increase the ulimit setting for locked memory).
+All translation units must select the same setting to avoid undefined
+behavior.
+
@opindex fopenacc
@cindex OpenACC accelerator programming
@item -fopenacc
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v3 4/6] openmp: -foffload-memory=pinned
2023-12-11 17:03 [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc Andrew Stubbs
` (2 preceding siblings ...)
2023-12-11 17:04 ` [PATCH v3 3/6] openmp: Add -foffload-memory Andrew Stubbs
@ 2023-12-11 17:04 ` Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 5/6] libgomp, nvptx: Cuda pinned memory Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 6/6] libgomp: fine-grained pinned memory allocator Andrew Stubbs
5 siblings, 0 replies; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-11 17:04 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 1745 bytes --]
Implement the -foffload-memory=pinned option such that libgomp is
instructed to enable fully-pinned memory at start-up. The option is
intended to provide a performance boost to certain offload programs without
modifying the code.
This feature only works on Linux, at present, and simply calls mlockall to
enable always-on memory pinning. It requires that the ulimit feature is
set high enough to accommodate all the program's memory usage.
In this mode the ompx_pinned_memory_alloc feature is disabled as it is not
needed and may conflict.
gcc/ChangeLog:
* omp-builtins.def (BUILT_IN_GOMP_ENABLE_PINNED_MODE): New.
* omp-low.cc (omp_enable_pinned_mode): New function.
(execute_lower_omp): Call omp_enable_pinned_mode.
libgomp/ChangeLog:
* config/linux/allocator.c (always_pinned_mode): New variable.
(GOMP_enable_pinned_mode): New function.
(linux_memspace_alloc): Disable pinning when always_pinned_mode set.
(linux_memspace_calloc): Likewise.
(linux_memspace_free): Likewise.
(linux_memspace_realloc): Likewise.
* libgomp.map: Add GOMP_enable_pinned_mode.
* testsuite/libgomp.c/alloc-pinned-7.c: New test.
* testsuite/libgomp.c-c++-common/alloc-pinned-1.c: New test.
---
gcc/omp-builtins.def | 3 +
gcc/omp-low.cc | 66 +++++++++++++++++++
libgomp/config/linux/allocator.c | 26 ++++++++
libgomp/libgomp.map | 1 +
.../libgomp.c-c++-common/alloc-pinned-1.c | 28 ++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-7.c | 63 ++++++++++++++++++
6 files changed, 187 insertions(+)
create mode 100644 libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-7.c
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0004-openmp-foffload-memory-pinned.patch --]
[-- Type: text/x-patch; name="v3-0004-openmp-foffload-memory-pinned.patch", Size: 7175 bytes --]
diff --git a/gcc/omp-builtins.def b/gcc/omp-builtins.def
index ed78d49d205..54ea7380722 100644
--- a/gcc/omp-builtins.def
+++ b/gcc/omp-builtins.def
@@ -473,3 +473,6 @@ DEF_GOMP_BUILTIN (BUILT_IN_GOMP_WARNING, "GOMP_warning",
BT_FN_VOID_CONST_PTR_SIZE, ATTR_NOTHROW_LEAF_LIST)
DEF_GOMP_BUILTIN (BUILT_IN_GOMP_ERROR, "GOMP_error",
BT_FN_VOID_CONST_PTR_SIZE, ATTR_COLD_NORETURN_NOTHROW_LEAF_LIST)
+DEF_GOMP_BUILTIN (BUILT_IN_GOMP_ENABLE_PINNED_MODE,
+ "GOMP_enable_pinned_mode",
+ BT_FN_VOID, ATTR_NOTHROW_LIST)
diff --git a/gcc/omp-low.cc b/gcc/omp-low.cc
index dd802ca37a6..455c5897577 100644
--- a/gcc/omp-low.cc
+++ b/gcc/omp-low.cc
@@ -14592,6 +14592,68 @@ lower_omp (gimple_seq *body, omp_context *ctx)
input_location = saved_location;
}
+/* Emit a constructor function to enable -foffload-memory=pinned
+ at runtime. Libgomp handles the OS mode setting, but we need to trigger
+ it by calling GOMP_enable_pinned mode before the program proper runs. */
+
+static void
+omp_enable_pinned_mode ()
+{
+ static bool visited = false;
+ if (visited)
+ return;
+ visited = true;
+
+ /* Create a new function like this:
+
+ static void __attribute__((constructor))
+ __set_pinned_mode ()
+ {
+ GOMP_enable_pinned_mode ();
+ }
+ */
+
+ tree name = get_identifier ("__set_pinned_mode");
+ tree voidfntype = build_function_type_list (void_type_node, NULL_TREE);
+ tree decl = build_decl (UNKNOWN_LOCATION, FUNCTION_DECL, name, voidfntype);
+
+ TREE_STATIC (decl) = 1;
+ TREE_USED (decl) = 1;
+ DECL_ARTIFICIAL (decl) = 1;
+ DECL_IGNORED_P (decl) = 0;
+ TREE_PUBLIC (decl) = 0;
+ DECL_UNINLINABLE (decl) = 1;
+ DECL_EXTERNAL (decl) = 0;
+ DECL_CONTEXT (decl) = NULL_TREE;
+ DECL_INITIAL (decl) = make_node (BLOCK);
+ BLOCK_SUPERCONTEXT (DECL_INITIAL (decl)) = decl;
+ DECL_STATIC_CONSTRUCTOR (decl) = 1;
+ DECL_ATTRIBUTES (decl) = tree_cons (get_identifier ("constructor"),
+ NULL_TREE, NULL_TREE);
+
+ tree t = build_decl (UNKNOWN_LOCATION, RESULT_DECL, NULL_TREE,
+ void_type_node);
+ DECL_ARTIFICIAL (t) = 1;
+ DECL_IGNORED_P (t) = 1;
+ DECL_CONTEXT (t) = decl;
+ DECL_RESULT (decl) = t;
+
+ push_struct_function (decl);
+ init_tree_ssa (cfun);
+
+ tree calldecl = builtin_decl_explicit (BUILT_IN_GOMP_ENABLE_PINNED_MODE);
+ gcall *call = gimple_build_call (calldecl, 0);
+
+ gimple_seq seq = NULL;
+ gimple_seq_add_stmt (&seq, call);
+ gimple_set_body (decl, gimple_build_bind (NULL_TREE, seq, NULL));
+
+ cfun->function_end_locus = UNKNOWN_LOCATION;
+ cfun->curr_properties |= PROP_gimple_any;
+ pop_cfun ();
+ cgraph_node::add_new_function (decl, true);
+}
+
/* Main entry point. */
static unsigned int
@@ -14648,6 +14710,10 @@ execute_lower_omp (void)
for (auto task_stmt : task_cpyfns)
finalize_task_copyfn (task_stmt);
task_cpyfns.release ();
+
+ if (flag_offload_memory == OFFLOAD_MEMORY_PINNED)
+ omp_enable_pinned_mode ();
+
return 0;
}
diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c
index 269d0d607d8..57278b1af91 100644
--- a/libgomp/config/linux/allocator.c
+++ b/libgomp/config/linux/allocator.c
@@ -51,11 +51,28 @@
#include <string.h>
#include "libgomp.h"
+static bool always_pinned_mode = false;
+
+/* This function is called by the compiler when -foffload-memory=pinned
+ is used. */
+
+void
+GOMP_enable_pinned_mode ()
+{
+ if (mlockall (MCL_CURRENT | MCL_FUTURE) != 0)
+ gomp_error ("failed to pin all memory (ulimit too low?)");
+ else
+ always_pinned_mode = true;
+}
+
static void *
linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin)
{
(void)memspace;
+ /* Explicit pinning may not be required. */
+ pin = pin && !always_pinned_mode;
+
if (pin)
{
/* Note that mmap always returns zeroed memory and is therefore also a
@@ -82,6 +99,9 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin)
static void *
linux_memspace_calloc (omp_memspace_handle_t memspace, size_t size, int pin)
{
+ /* Explicit pinning may not be required. */
+ pin = pin && !always_pinned_mode;
+
if (pin)
return linux_memspace_alloc (memspace, size, pin);
else
@@ -94,6 +114,9 @@ linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size,
{
(void)memspace;
+ /* Explicit pinning may not be required. */
+ pin = pin && !always_pinned_mode;
+
if (pin)
munmap (addr, size);
else
@@ -104,6 +127,9 @@ static void *
linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr,
size_t oldsize, size_t size, int oldpin, int pin)
{
+ /* Explicit pinning may not be required. */
+ pin = pin && !always_pinned_mode;
+
if (oldpin && pin)
{
void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE);
diff --git a/libgomp/libgomp.map b/libgomp/libgomp.map
index 90c401453b2..eb8cdc3a8c6 100644
--- a/libgomp/libgomp.map
+++ b/libgomp/libgomp.map
@@ -406,6 +406,7 @@ GOMP_5.0.1 {
global:
GOMP_alloc;
GOMP_free;
+ GOMP_enable_pinned_mode;
} GOMP_5.0;
GOMP_5.1 {
diff --git a/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c
new file mode 100644
index 00000000000..e0e08019bff
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c
@@ -0,0 +1,28 @@
+/* { dg-do run } */
+/* { dg-additional-options "-foffload-memory=pinned" } */
+/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+
+#if __cplusplus
+#define EXTERNC extern "C"
+#else
+#define EXTERNC
+#endif
+
+/* Intercept the libgomp initialization call to check it happens. */
+
+int good = 0;
+
+EXTERNC void
+GOMP_enable_pinned_mode ()
+{
+ good = 1;
+}
+
+int
+main ()
+{
+ if (!good)
+ __builtin_exit (1);
+
+ return 0;
+}
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-7.c b/libgomp/testsuite/libgomp.c/alloc-pinned-7.c
new file mode 100644
index 00000000000..350bcd36c5a
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-7.c
@@ -0,0 +1,63 @@
+/* { dg-do run } */
+/* { dg-additional-options "-foffload-memory=pinned" } */
+
+/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+
+/* Test that -foffload-memory=pinned works. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+#else
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+
+#define mlockall(...) 0
+#endif
+
+#include <omp.h>
+
+int
+main ()
+{
+ // Sanity check
+ if (get_pinned_mem () == 0)
+ {
+ /* -foffload-memory=pinned has failed, but maybe that's because
+ isufficient pinned memory was available. */
+ if (mlockall (MCL_CURRENT | MCL_FUTURE) == 0)
+ abort ();
+ }
+
+ return 0;
+}
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v3 5/6] libgomp, nvptx: Cuda pinned memory
2023-12-11 17:03 [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc Andrew Stubbs
` (3 preceding siblings ...)
2023-12-11 17:04 ` [PATCH v3 4/6] openmp: -foffload-memory=pinned Andrew Stubbs
@ 2023-12-11 17:04 ` Andrew Stubbs
2023-12-12 9:10 ` Tobias Burnus
2023-12-11 17:04 ` [PATCH v3 6/6] libgomp: fine-grained pinned memory allocator Andrew Stubbs
5 siblings, 1 reply; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-11 17:04 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]
Use Cuda to pin memory, instead of Linux mlock, when available.
There are two advantages: firstly, this gives a significant speed boost for
NVPTX offloading, and secondly, it side-steps the usual OS ulimit/rlimit
setting.
The design adds a device independent plugin API for allocating pinned memory,
and then implements it for NVPTX. At present, the other supported devices do
not have equivalent capabilities (or requirements).
libgomp/ChangeLog:
* config/linux/allocator.c: Include assert.h.
(using_device_for_page_locked): New variable.
(linux_memspace_alloc): Add init0 parameter. Support device pinning.
(linux_memspace_calloc): Set init0 to true.
(linux_memspace_free): Support device pinning.
(linux_memspace_realloc): Support device pinning.
(MEMSPACE_ALLOC): Set init0 to false.
* libgomp-plugin.h
(GOMP_OFFLOAD_page_locked_host_alloc): New prototype.
(GOMP_OFFLOAD_page_locked_host_free): Likewise.
* libgomp.h (gomp_page_locked_host_alloc): Likewise.
(gomp_page_locked_host_free): Likewise.
(struct gomp_device_descr): Add page_locked_host_alloc_func and
page_locked_host_free_func.
* libgomp.texi: Adjust the docs for the pinned trait.
* libgomp_g.h (GOMP_enable_pinned_mode): New prototype.
* plugin/plugin-nvptx.c
(GOMP_OFFLOAD_page_locked_host_alloc): New function.
(GOMP_OFFLOAD_page_locked_host_free): Likewise.
* target.c (device_for_page_locked): New variable.
(get_device_for_page_locked): New function.
(gomp_page_locked_host_alloc): Likewise.
(gomp_page_locked_host_free): Likewise.
(gomp_load_plugin_for_device): Add page_locked_host_alloc and
page_locked_host_free.
* testsuite/libgomp.c/alloc-pinned-1.c: Change expectations for NVPTX
devices.
* testsuite/libgomp.c/alloc-pinned-2.c: Likewise.
* testsuite/libgomp.c/alloc-pinned-3.c: Likewise.
* testsuite/libgomp.c/alloc-pinned-4.c: Likewise.
* testsuite/libgomp.c/alloc-pinned-5.c: Likewise.
* testsuite/libgomp.c/alloc-pinned-6.c: Likewise.
Co-Authored-By: Thomas Schwinge <thomas@codesourcery.com>
---
libgomp/config/linux/allocator.c | 137 ++++++++++++++-----
libgomp/libgomp-plugin.h | 2 +
libgomp/libgomp.h | 4 +
libgomp/libgomp.texi | 11 +-
libgomp/libgomp_g.h | 1 +
libgomp/plugin/plugin-nvptx.c | 42 ++++++
libgomp/target.c | 136 ++++++++++++++++++
libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 26 ++++
libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 26 ++++
libgomp/testsuite/libgomp.c/alloc-pinned-3.c | 45 +++++-
libgomp/testsuite/libgomp.c/alloc-pinned-4.c | 44 +++++-
libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 26 ++++
libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 35 ++++-
13 files changed, 487 insertions(+), 48 deletions(-)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0005-libgomp-nvptx-Cuda-pinned-memory.patch --]
[-- Type: text/x-patch; name="v3-0005-libgomp-nvptx-Cuda-pinned-memory.patch", Size: 29819 bytes --]
diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c
index 57278b1af91..8d681b5ec50 100644
--- a/libgomp/config/linux/allocator.c
+++ b/libgomp/config/linux/allocator.c
@@ -36,6 +36,11 @@
/* Implement malloc routines that can handle pinned memory on Linux.
+ Given that pinned memory is typically used to help host <-> device memory
+ transfers, we attempt to allocate such memory using a device (really:
+ libgomp plugin), but fall back to mmap plus mlock if no suitable device is
+ available.
+
It's possible to use mlock on any heap memory, but using munlock is
problematic if there are multiple pinned allocations on the same page.
Tracking all that manually would be possible, but adds overhead. This may
@@ -49,6 +54,7 @@
#define _GNU_SOURCE
#include <sys/mman.h>
#include <string.h>
+#include <assert.h>
#include "libgomp.h"
static bool always_pinned_mode = false;
@@ -65,45 +71,87 @@ GOMP_enable_pinned_mode ()
always_pinned_mode = true;
}
+static int using_device_for_page_locked
+ = /* uninitialized */ -1;
+
static void *
-linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin)
+linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin,
+ bool init0)
{
- (void)memspace;
+ gomp_debug (0, "%s: memspace=%llu, size=%llu, pin=%d, init0=%d\n",
+ __FUNCTION__, (unsigned long long) memspace,
+ (unsigned long long) size, pin, init0);
+
+ void *addr;
/* Explicit pinning may not be required. */
pin = pin && !always_pinned_mode;
if (pin)
{
- /* Note that mmap always returns zeroed memory and is therefore also a
- suitable implementation of calloc. */
- void *addr = mmap (NULL, size, PROT_READ | PROT_WRITE,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- if (addr == MAP_FAILED)
- return NULL;
-
- if (mlock (addr, size))
+ int using_device
+ = __atomic_load_n (&using_device_for_page_locked,
+ MEMMODEL_RELAXED);
+ gomp_debug (0, " using_device=%d\n",
+ using_device);
+ if (using_device != 0)
{
- gomp_debug (0, "libgomp: failed to pin %ld bytes of"
- " memory (ulimit too low?)\n", size);
- munmap (addr, size);
- return NULL;
+ using_device = gomp_page_locked_host_alloc (&addr, size);
+ int using_device_old
+ = __atomic_exchange_n (&using_device_for_page_locked,
+ using_device, MEMMODEL_RELAXED);
+ gomp_debug (0, " using_device=%d, using_device_old=%d\n",
+ using_device, using_device_old);
+ assert (using_device_old == -1
+ /* We shouldn't have concurrently changed our mind. */
+ || using_device_old == using_device);
+ }
+ if (using_device == 0)
+ {
+ gomp_debug (0, " mmap\n");
+ addr = mmap (NULL, size, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (addr == MAP_FAILED)
+ addr = NULL;
+ else
+ {
+ /* 'mmap' zero-initializes. */
+ init0 = false;
+
+ gomp_debug (0, " mlock\n");
+ if (mlock (addr, size))
+ {
+ gomp_debug (0, "libgomp: failed to pin %ld bytes of"
+ " memory (ulimit too low?)\n", size);
+ munmap (addr, size);
+ addr = NULL;
+ }
+ }
}
-
- return addr;
}
else
- return malloc (size);
+ addr = malloc (size);
+
+ if (addr && init0)
+ {
+ gomp_debug (0, " init0\n");
+ memset (addr, 0, size);
+ }
+
+ return addr;
}
static void *
linux_memspace_calloc (omp_memspace_handle_t memspace, size_t size, int pin)
{
+ gomp_debug (0, "%s: memspace=%llu, size=%llu, pin=%d\n",
+ __FUNCTION__, (unsigned long long) memspace, (unsigned long long) size, pin);
+
/* Explicit pinning may not be required. */
pin = pin && !always_pinned_mode;
if (pin)
- return linux_memspace_alloc (memspace, size, pin);
+ return linux_memspace_alloc (memspace, size, pin, true);
else
return calloc (1, size);
}
@@ -112,13 +160,25 @@ static void
linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size,
int pin)
{
- (void)memspace;
+ gomp_debug (0, "%s: memspace=%llu, addr=%p, size=%llu, pin=%d\n",
+ __FUNCTION__, (unsigned long long) memspace, addr, (unsigned long long) size, pin);
/* Explicit pinning may not be required. */
pin = pin && !always_pinned_mode;
if (pin)
- munmap (addr, size);
+ {
+ int using_device
+ = __atomic_load_n (&using_device_for_page_locked,
+ MEMMODEL_RELAXED);
+ gomp_debug (0, " using_device=%d\n",
+ using_device);
+ if (using_device == 1)
+ gomp_page_locked_host_free (addr);
+ else
+ /* 'munlock'ing is implicit with following 'munmap'. */
+ munmap (addr, size);
+ }
else
free (addr);
}
@@ -127,11 +187,25 @@ static void *
linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr,
size_t oldsize, size_t size, int oldpin, int pin)
{
+ gomp_debug (0, "%s: memspace=%llu, addr=%p, oldsize=%llu, size=%llu, oldpin=%d, pin=%d\n",
+ __FUNCTION__, (unsigned long long) memspace, addr, (unsigned long long) oldsize, (unsigned long long) size, oldpin, pin);
+
/* Explicit pinning may not be required. */
pin = pin && !always_pinned_mode;
if (oldpin && pin)
{
+ /* We can only expect to be able to just 'mremap' if not using a device
+ for page-locked memory. */
+ int using_device
+ = __atomic_load_n (&using_device_for_page_locked,
+ MEMMODEL_RELAXED);
+ gomp_debug (0, " using_device=%d\n",
+ using_device);
+ if (using_device != 0)
+ goto manual_realloc;
+
+ gomp_debug (0, " mremap\n");
void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE);
if (newaddr == MAP_FAILED)
return NULL;
@@ -139,18 +213,19 @@ linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr,
return newaddr;
}
else if (oldpin || pin)
- {
- void *newaddr = linux_memspace_alloc (memspace, size, pin);
- if (newaddr)
- {
- memcpy (newaddr, addr, oldsize < size ? oldsize : size);
- linux_memspace_free (memspace, addr, oldsize, oldpin);
- }
-
- return newaddr;
- }
+ goto manual_realloc;
else
return realloc (addr, size);
+
+manual_realloc:;
+ void *newaddr = linux_memspace_alloc (memspace, size, pin, false);
+ if (newaddr)
+ {
+ memcpy (newaddr, addr, oldsize < size ? oldsize : size);
+ linux_memspace_free (memspace, addr, oldsize, oldpin);
+ }
+
+ return newaddr;
}
static int
@@ -161,7 +236,7 @@ linux_memspace_validate (omp_memspace_handle_t, unsigned, int)
}
#define MEMSPACE_ALLOC(MEMSPACE, SIZE, PIN) \
- linux_memspace_alloc (MEMSPACE, SIZE, PIN)
+ linux_memspace_alloc (MEMSPACE, SIZE, PIN, false)
#define MEMSPACE_CALLOC(MEMSPACE, SIZE, PIN) \
linux_memspace_calloc (MEMSPACE, SIZE, PIN)
#define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN) \
diff --git a/libgomp/libgomp-plugin.h b/libgomp/libgomp-plugin.h
index 3ce032c5cc0..319818de121 100644
--- a/libgomp/libgomp-plugin.h
+++ b/libgomp/libgomp-plugin.h
@@ -139,6 +139,8 @@ extern int GOMP_OFFLOAD_load_image (int, unsigned, const void *,
extern bool GOMP_OFFLOAD_unload_image (int, unsigned, const void *);
extern void *GOMP_OFFLOAD_alloc (int, size_t);
extern bool GOMP_OFFLOAD_free (int, void *);
+extern bool GOMP_OFFLOAD_page_locked_host_alloc (void **, size_t);
+extern bool GOMP_OFFLOAD_page_locked_host_free (void *);
extern bool GOMP_OFFLOAD_dev2host (int, void *, const void *, size_t);
extern bool GOMP_OFFLOAD_host2dev (int, void *, const void *, size_t);
extern bool GOMP_OFFLOAD_dev2dev (int, void *, const void *, size_t);
diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h
index fa29f428976..8c9e50d88fd 100644
--- a/libgomp/libgomp.h
+++ b/libgomp/libgomp.h
@@ -1128,6 +1128,8 @@ extern int gomp_get_num_devices (void);
extern bool gomp_target_task_fn (void *);
extern void gomp_target_rev (uint64_t, uint64_t, uint64_t, uint64_t, uint64_t,
int, struct goacc_asyncqueue *);
+extern bool gomp_page_locked_host_alloc (void **, size_t);
+extern void gomp_page_locked_host_free (void *);
/* Splay tree definitions. */
typedef struct splay_tree_node_s *splay_tree_node;
@@ -1406,6 +1408,8 @@ struct gomp_device_descr
__typeof (GOMP_OFFLOAD_unload_image) *unload_image_func;
__typeof (GOMP_OFFLOAD_alloc) *alloc_func;
__typeof (GOMP_OFFLOAD_free) *free_func;
+ __typeof (GOMP_OFFLOAD_page_locked_host_alloc) *page_locked_host_alloc_func;
+ __typeof (GOMP_OFFLOAD_page_locked_host_free) *page_locked_host_free_func;
__typeof (GOMP_OFFLOAD_dev2host) *dev2host_func;
__typeof (GOMP_OFFLOAD_host2dev) *host2dev_func;
__typeof (GOMP_OFFLOAD_memcpy2d) *memcpy2d_func;
diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi
index 1f73c49c9b8..a3a49d59409 100644
--- a/libgomp/libgomp.texi
+++ b/libgomp/libgomp.texi
@@ -5756,8 +5756,9 @@ a @code{nearest} allocation.
Additional notes regarding the traits:
@itemize
-@item The @code{pinned} trait is supported on Linux hosts, but is subject to
- the OS @code{ulimit}/@code{rlimit} locked memory settings.
+@item The @code{pinned} trait is supported on Linux hosts, but is usually
+ subject to the OS @code{ulimit}/@code{rlimit} locked memory settings (see
+ @ref{Offload-Target Specifics} for exceptions).
@item The default for the @code{pool_size} trait is no pool and for every
(re)allocation the associated library routine is called, which might
internally use a memory pool.
@@ -5937,6 +5938,12 @@ The implementation remark:
@code{omp_thread_mem_alloc}, all use low-latency memory as first
preference, and fall back to main graphics memory when the low-latency
pool is exhausted.
+@item Pinned memory allocated using @code{omp_alloc} with the
+ @code{ompx_pinned_mem_alloc} allocator or the @code{pinned} trait is
+ obtained via the CUDA API when an NVPTX device is present. This provides
+ a performance boost for NVPTX offload code and also allows unlimited use
+ of pinned memory regardless of the OS @code{ulimit}/@code{rlimit}
+ settings.
@end itemize
diff --git a/libgomp/libgomp_g.h b/libgomp/libgomp_g.h
index 95046312ae9..1773f500c94 100644
--- a/libgomp/libgomp_g.h
+++ b/libgomp/libgomp_g.h
@@ -368,6 +368,7 @@ extern void GOMP_teams_reg (void (*) (void *), void *, unsigned, unsigned,
extern void *GOMP_alloc (size_t, size_t, uintptr_t);
extern void GOMP_free (void *, uintptr_t);
+extern void GOMP_enable_pinned_mode (void);
/* error.c */
diff --git a/libgomp/plugin/plugin-nvptx.c b/libgomp/plugin/plugin-nvptx.c
index d4a254ed4f0..3fb6650a393 100644
--- a/libgomp/plugin/plugin-nvptx.c
+++ b/libgomp/plugin/plugin-nvptx.c
@@ -1608,6 +1608,48 @@ GOMP_OFFLOAD_free (int ord, void *ptr)
&& nvptx_free (ptr, ptx_devices[ord]));
}
+bool
+GOMP_OFFLOAD_page_locked_host_alloc (void **ptr, size_t size)
+{
+ GOMP_PLUGIN_debug (0, "nvptx %s: ptr=%p, size=%llu\n",
+ __FUNCTION__, ptr, (unsigned long long) size);
+
+ if (size == 0)
+ {
+ /* Special case to ensure omp_alloc specification compliance. */
+ *ptr = NULL;
+ GOMP_PLUGIN_debug (0, " -> *ptr=null\n");
+ return true;
+ }
+
+ CUresult r;
+
+ unsigned int flags = 0;
+ /* Given 'CU_DEVICE_ATTRIBUTE_UNIFIED_ADDRESSING', we don't need
+ 'flags |= CU_MEMHOSTALLOC_PORTABLE;' here. */
+ r = CUDA_CALL_NOCHECK (cuMemHostAlloc, ptr, size, flags);
+ if (r == CUDA_ERROR_OUT_OF_MEMORY)
+ *ptr = NULL;
+ else if (r != CUDA_SUCCESS)
+ {
+ GOMP_PLUGIN_error ("cuMemHostAlloc error: %s", cuda_error (r));
+ return false;
+ }
+ GOMP_PLUGIN_debug (0, " -> *ptr=%p\n",
+ *ptr);
+ return true;
+}
+
+bool
+GOMP_OFFLOAD_page_locked_host_free (void *ptr)
+{
+ GOMP_PLUGIN_debug (0, "nvptx %s: ptr=%p\n",
+ __FUNCTION__, ptr);
+
+ CUDA_CALL (cuMemFreeHost, ptr);
+ return true;
+}
+
void
GOMP_OFFLOAD_openacc_exec (void (*fn) (void *),
size_t mapnum __attribute__((unused)),
diff --git a/libgomp/target.c b/libgomp/target.c
index f30c20255d3..d7c9ab780aa 100644
--- a/libgomp/target.c
+++ b/libgomp/target.c
@@ -4345,6 +4345,140 @@ omp_target_free (void *device_ptr, int device_num)
gomp_mutex_unlock (&devicep->lock);
}
+/* Device (really: libgomp plugin) to use for paged-locked memory. We
+ assume there is either none or exactly one such device for the lifetime of
+ the process. */
+
+static struct gomp_device_descr *device_for_page_locked
+ = /* uninitialized */ (void *) -1;
+
+static struct gomp_device_descr *
+get_device_for_page_locked (void)
+{
+ gomp_debug (0, "%s\n",
+ __FUNCTION__);
+
+ struct gomp_device_descr *device;
+#ifdef HAVE_SYNC_BUILTINS
+ device
+ = __atomic_load_n (&device_for_page_locked, MEMMODEL_RELAXED);
+ if (device == (void *) -1)
+ {
+ gomp_debug (0, " init\n");
+
+ gomp_init_targets_once ();
+
+ device = NULL;
+ for (int i = 0; i < num_devices; ++i)
+ {
+ gomp_debug (0, " i=%d, target_id=%d\n",
+ i, devices[i].target_id);
+
+ /* We consider only the first device of potentially several of the
+ same type as this functionality is not specific to an individual
+ offloading device, but instead relates to the host-side
+ implementation of the respective offloading implementation. */
+ if (devices[i].target_id != 0)
+ continue;
+
+ if (!devices[i].page_locked_host_alloc_func)
+ continue;
+
+ gomp_debug (0, " found device: %p (%s)\n",
+ &devices[i], devices[i].name);
+ if (device)
+ gomp_fatal ("Unclear how %s and %s libgomp plugins may"
+ " simultaneously provide functionality"
+ " for page-locked memory",
+ device->name, devices[i].name);
+ else
+ device = &devices[i];
+ }
+
+ struct gomp_device_descr *device_old
+ = __atomic_exchange_n (&device_for_page_locked, device,
+ MEMMODEL_RELAXED);
+ gomp_debug (0, " old device_for_page_locked: %p\n",
+ device_old);
+ assert (device_old == (void *) -1
+ /* We shouldn't have concurrently found a different or no
+ device. */
+ || device_old == device);
+ }
+#else /* !HAVE_SYNC_BUILTINS */
+ gomp_debug (0, " not implemented for '!HAVE_SYNC_BUILTINS'\n");
+ (void) &device_for_page_locked;
+ device = NULL;
+#endif /* HAVE_SYNC_BUILTINS */
+
+ gomp_debug (0, " -> device=%p (%s)\n",
+ device, device ? device->name : "[none]");
+ return device;
+}
+
+/* Allocate page-locked host memory.
+ Returns whether we have a device capable of that. */
+
+attribute_hidden bool
+gomp_page_locked_host_alloc (void **ptr, size_t size)
+{
+ gomp_debug (0, "%s: ptr=%p, size=%llu\n",
+ __FUNCTION__, ptr, (unsigned long long) size);
+
+ struct gomp_device_descr *device = get_device_for_page_locked ();
+ gomp_debug (0, " device=%p (%s)\n",
+ device, device ? device->name : "[none]");
+ if (device)
+ {
+ gomp_mutex_lock (&device->lock);
+ if (device->state == GOMP_DEVICE_UNINITIALIZED)
+ gomp_init_device (device);
+ else if (device->state == GOMP_DEVICE_FINALIZED)
+ {
+ gomp_mutex_unlock (&device->lock);
+ gomp_fatal ("Device %s used for for page-locked memory is finalized",
+ device->name);
+ }
+ gomp_mutex_unlock (&device->lock);
+
+ if (!device->page_locked_host_alloc_func (ptr, size))
+ gomp_fatal ("Failed to allocate page-locked host memory"
+ " via %s libgomp plugin",
+ device->name);
+ }
+ return device != NULL;
+}
+
+/* Free page-locked host memory.
+ This must only be called if 'gomp_page_locked_host_alloc' returned
+ 'true'. */
+
+attribute_hidden void
+gomp_page_locked_host_free (void *ptr)
+{
+ gomp_debug (0, "%s: ptr=%p\n",
+ __FUNCTION__, ptr);
+
+ struct gomp_device_descr *device = get_device_for_page_locked ();
+ gomp_debug (0, " device=%p (%s)\n",
+ device, device ? device->name : "[none]");
+ assert (device);
+
+ gomp_mutex_lock (&device->lock);
+ assert (device->state != GOMP_DEVICE_UNINITIALIZED);
+ if (device->state == GOMP_DEVICE_FINALIZED)
+ {
+ gomp_mutex_unlock (&device->lock);
+ return;
+ }
+ gomp_mutex_unlock (&device->lock);
+
+ if (!device->page_locked_host_free_func (ptr))
+ gomp_fatal ("Failed to free page-locked host memory"
+ " via %s libgomp plugin",
+ device->name);
+}
+
int
omp_target_is_present (const void *ptr, int device_num)
{
@@ -5067,6 +5201,8 @@ gomp_load_plugin_for_device (struct gomp_device_descr *device,
DLSYM (unload_image);
DLSYM (alloc);
DLSYM (free);
+ DLSYM_OPT (page_locked_host_alloc, page_locked_host_alloc);
+ DLSYM_OPT (page_locked_host_free, page_locked_host_free);
DLSYM (dev2host);
DLSYM (host2dev);
DLSYM_OPT (memcpy2d, memcpy2d);
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c
index e17a21f0a6c..2da660b17ec 100644
--- a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c
@@ -2,6 +2,8 @@
/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */
+
/* Test that pinned memory works. */
#include <stdio.h>
@@ -68,10 +70,16 @@ verify0 (char *p, size_t s)
int
main ()
{
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* Go big or go home.
+ The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */
+ const int SIZE = 40 * 1024 * 1024;
+#else
/* Allocate at least a page each time, allowing space for overhead,
but stay within the ulimit. */
const int SIZE = PAGE_SIZE - 128;
CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures
+#endif
const omp_alloctrait_t traits[] = {
{ omp_atk_pinned, 1 }
@@ -93,21 +101,39 @@ main ()
abort ();
int amount = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount != 0)
+ abort ();
+#else
if (amount == 0)
abort ();
+#endif
p = omp_realloc (p, SIZE * 2, allocator, allocator);
int amount2 = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount2 != 0)
+ abort ();
+#else
if (amount2 <= amount)
abort ();
+#endif
/* SIZE*2 ensures that it doesn't slot into the space possibly
vacated by realloc. */
p = omp_calloc (1, SIZE * 2, allocator);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (get_pinned_mem () != 0)
+ abort ();
+#else
if (get_pinned_mem () <= amount2)
abort ();
+#endif
verify0 (p, SIZE * 2);
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c
index 3cf322cfbc8..5dd765d24c5 100644
--- a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c
@@ -2,6 +2,8 @@
/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */
+
/* Test that pinned memory works (pool_size code path). */
#include <stdio.h>
@@ -68,10 +70,16 @@ verify0 (char *p, size_t s)
int
main ()
{
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* Go big or go home.
+ The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */
+ const int SIZE = 40 * 1024 * 1024;
+#else
/* Allocate at least a page each time, allowing space for overhead,
but stay within the ulimit. */
const int SIZE = PAGE_SIZE - 128;
CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures
+#endif
const omp_alloctrait_t traits[] = {
{ omp_atk_pinned, 1 },
@@ -94,16 +102,28 @@ main ()
abort ();
int amount = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount != 0)
+ abort ();
+#else
if (amount == 0)
abort ();
+#endif
p = omp_realloc (p, SIZE * 2, allocator, allocator);
if (!p)
abort ();
int amount2 = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount2 != 0)
+ abort ();
+#else
if (amount2 <= amount)
abort ();
+#endif
/* SIZE*2 ensures that it doesn't slot into the space possibly
vacated by realloc. */
@@ -111,8 +131,14 @@ main ()
if (!p)
abort ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (get_pinned_mem () != 0)
+ abort ();
+#else
if (get_pinned_mem () <= amount2)
abort ();
+#endif
verify0 (p, SIZE * 2);
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-3.c b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c
index 53e4720cc9c..b17daf7e222 100644
--- a/libgomp/testsuite/libgomp.c/alloc-pinned-3.c
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c
@@ -1,5 +1,7 @@
/* { dg-do run } */
+/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */
+
/* Test that pinned memory fails correctly. */
#include <stdio.h>
@@ -75,8 +77,15 @@ verify0 (char *p, size_t s)
int
main ()
{
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* Go big or go home.
+ The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */
+ const int SIZE = 40 * 1024 * 1024;
+#else
/* This needs to be large enough to cover multiple pages. */
const int SIZE = PAGE_SIZE * 4;
+#endif
+ const int PIN_LIMIT = PAGE_SIZE * 2;
/* Pinned memory, no fallback. */
const omp_alloctrait_t traits1[] = {
@@ -101,23 +110,34 @@ main ()
#endif
/* Ensure that the limit is smaller than the allocation. */
- set_pin_limit (SIZE / 2);
+ set_pin_limit (PIN_LIMIT);
// Sanity check
if (get_pinned_mem () != 0)
abort ();
- // Should fail
void *p1 = omp_alloc (SIZE, allocator1);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'.
+ if (!p1)
+ abort ();
+#else
+ // Should fail
if (p1)
abort ();
+#endif
- // Should fail
void *p2 = omp_calloc (1, SIZE, allocator1);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'.
+ if (!p2)
+ abort ();
+#else
+ // Should fail
if (p2)
abort ();
+#endif
- // Should fall back
void *p3 = omp_alloc (SIZE, allocator2);
if (!p3)
abort ();
@@ -128,16 +148,29 @@ main ()
abort ();
verify0 (p4, SIZE);
- // Should fail to realloc
void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc);
void *p5 = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'; does reallocate.
+ if (!notpinned || !p5 || p5 == notpinned)
+ abort ();
+#else
+ // Should fail to realloc
if (!notpinned || p5)
abort ();
+#endif
- // Should fall back to no realloc needed
+#ifdef OFFLOAD_DEVICE_NVPTX
+ void *p6 = omp_realloc (p5, SIZE, allocator2, allocator1);
+ // Does reallocate.
+ if (p5 == p6)
+ abort ();
+#else
void *p6 = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc);
+ // Should fall back to no realloc needed
if (p6 != notpinned)
abort ();
+#endif
// No memory should have been pinned
int amount = get_pinned_mem ();
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-4.c b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c
index 9d850c23e4b..aadd9249ba9 100644
--- a/libgomp/testsuite/libgomp.c/alloc-pinned-4.c
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c
@@ -1,5 +1,7 @@
/* { dg-do run } */
+/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */
+
/* Test that pinned memory fails correctly, pool_size code path. */
#include <stdio.h>
@@ -75,8 +77,15 @@ verify0 (char *p, size_t s)
int
main ()
{
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* Go big or go home.
+ The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */
+ const int SIZE = 40 * 1024 * 1024;
+#else
/* This needs to be large enough to cover multiple pages. */
const int SIZE = PAGE_SIZE * 4;
+#endif
+ const int PIN_LIMIT = PAGE_SIZE * 2;
/* Pinned memory, no fallback. */
const omp_alloctrait_t traits1[] = {
@@ -103,21 +112,33 @@ main ()
#endif
/* Ensure that the limit is smaller than the allocation. */
- set_pin_limit (SIZE / 2);
+ set_pin_limit (PIN_LIMIT);
// Sanity check
if (get_pinned_mem () != 0)
abort ();
- // Should fail
void *p = omp_alloc (SIZE, allocator1);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'.
+ if (!p)
+ abort ();
+#else
+ // Should fail
if (p)
abort ();
+#endif
- // Should fail
p = omp_calloc (1, SIZE, allocator1);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'.
+ if (!p)
+ abort ();
+#else
+ // Should fail
if (p)
abort ();
+#endif
// Should fall back
p = omp_alloc (SIZE, allocator2);
@@ -130,16 +151,29 @@ main ()
abort ();
verify0 (p, SIZE);
- // Should fail to realloc
void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc);
p = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'; does reallocate.
+ if (!notpinned || !p || p == notpinned)
+ abort ();
+#else
+ // Should fail to realloc
if (!notpinned || p)
abort ();
+#endif
- // Should fall back to no realloc needed
+#ifdef OFFLOAD_DEVICE_NVPTX
+ void *p_ = omp_realloc (p, SIZE, allocator2, allocator1);
+ // Does reallocate.
+ if (p_ == p)
+ abort ();
+#else
p = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc);
+ // Should fall back to no realloc needed
if (p != notpinned)
abort ();
+#endif
// No memory should have been pinned
int amount = get_pinned_mem ();
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c
index 18e6d20ca5b..95e4562537c 100644
--- a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c
@@ -2,6 +2,8 @@
/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */
+
/* Test that ompx_pinned_mem_alloc works. */
#include <stdio.h>
@@ -67,10 +69,16 @@ verify0 (char *p, size_t s)
int
main ()
{
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* Go big or go home.
+ The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */
+ const int SIZE = 40 * 1024 * 1024;
+#else
/* Allocate at least a page each time, allowing space for overhead,
but stay within the ulimit. */
const int SIZE = PAGE_SIZE - 128;
CHECK_SIZE (SIZE * 5);
+#endif
// Sanity check
if (get_pinned_mem () != 0)
@@ -81,21 +89,39 @@ main ()
abort ();
int amount = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount != 0)
+ abort ();
+#else
if (amount == 0)
abort ();
+#endif
p = omp_realloc (p, SIZE * 2, ompx_pinned_mem_alloc, ompx_pinned_mem_alloc);
int amount2 = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount2 != 0)
+ abort ();
+#else
if (amount2 <= amount)
abort ();
+#endif
/* SIZE*2 ensures that it doesn't slot into the space possibly
vacated by realloc. */
p = omp_calloc (1, SIZE * 2, ompx_pinned_mem_alloc);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (get_pinned_mem () != 0)
+ abort ();
+#else
if (get_pinned_mem () <= amount2)
abort ();
+#endif
verify0 (p, SIZE * 2);
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c
index f80a0264f97..aefb69a1d60 100644
--- a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c
@@ -1,5 +1,7 @@
/* { dg-do run } */
+/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */
+
/* Test that ompx_pinned_mem_alloc fails correctly. */
#include <stdio.h>
@@ -66,31 +68,56 @@ set_pin_limit ()
int
main ()
{
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* Go big or go home.
+ The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */
+ const int SIZE = 40 * 1024 * 1024;
+#else
/* Allocate at least a page each time, but stay within the ulimit. */
const int SIZE = PAGE_SIZE * 4;
+#endif
+ const int PIN_LIMIT = PAGE_SIZE*2;
/* Ensure that the limit is smaller than the allocation. */
- set_pin_limit (SIZE / 2);
+ set_pin_limit (PIN_LIMIT);
// Sanity check
if (get_pinned_mem () != 0)
abort ();
- // Should fail
void *p = omp_alloc (SIZE, ompx_pinned_mem_alloc);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'.
+ if (!p)
+ abort ();
+#else
+ // Should fail
if (p)
abort ();
+#endif
- // Should fail
p = omp_calloc (1, SIZE, ompx_pinned_mem_alloc);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'.
+ if (!p)
+ abort ();
+#else
+ // Should fail
if (p)
abort ();
+#endif
- // Should fail to realloc
void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc);
p = omp_realloc (notpinned, SIZE, ompx_pinned_mem_alloc, omp_default_mem_alloc);
+#ifdef OFFLOAD_DEVICE_NVPTX
+ // Doesn't care about 'set_pin_limit'; does reallocate.
+ if (!notpinned || !p || p == notpinned)
+ abort ();
+#else
+ // Should fail to realloc
if (!notpinned || p)
abort ();
+#endif
// No memory should have been pinned
int amount = get_pinned_mem ();
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v3 6/6] libgomp: fine-grained pinned memory allocator
2023-12-11 17:03 [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc Andrew Stubbs
` (4 preceding siblings ...)
2023-12-11 17:04 ` [PATCH v3 5/6] libgomp, nvptx: Cuda pinned memory Andrew Stubbs
@ 2023-12-11 17:04 ` Andrew Stubbs
5 siblings, 0 replies; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-11 17:04 UTC (permalink / raw)
To: gcc-patches
[-- Attachment #1: Type: text/plain, Size: 2705 bytes --]
This patch introduces a new custom memory allocator for use with pinned
memory (in the case where the Cuda allocator isn't available). In future,
this allocator will also be used for Unified Shared Memory. Both memories
are incompatible with the system malloc because allocated memory cannot
share a page with memory allocated for other purposes.
This means that small allocations will no longer consume an entire page of
pinned memory. Unfortunately, it also means that pinned memory pages will
never be unmapped (although they may be reused).
The implementation is not perfect; there are various corner cases (especially
related to extending onto new pages) where allocations and reallocations may
be sub-optimal, but it should still be a step forward in support for small
allocations.
I have considered using libmemkind's "fixed" memory but rejected it for three
reasons: 1) libmemkind may not always be present at runtime, 2) there's no
currently documented means to extend a "fixed" kind one page at a time
(although the code appears to have an undocumented function that may do the
job, and/or extending libmemkind to support the MAP_LOCKED mmap flag with its
regular kinds would be straight-forward), 3) USM benefits from having the
metadata located in different memory and using an external implementation makes
it hard to guarantee this.
libgomp/ChangeLog:
* Makefile.am (libgomp_la_SOURCES): Add usmpin-allocator.c.
* Makefile.in: Regenerate.
* config/linux/allocator.c: Include unistd.h.
(pin_ctx): New variable.
(ctxlock): New variable.
(linux_init_pin_ctx): New function.
(linux_memspace_alloc): Use usmpin-allocator for pinned memory.
(linux_memspace_free): Likewise.
(linux_memspace_realloc): Likewise.
* libgomp.h (usmpin_init_context): New prototype.
(usmpin_register_memory): New prototype.
(usmpin_alloc): New prototype.
(usmpin_free): New prototype.
(usmpin_realloc): New prototype.
* testsuite/libgomp.c/alloc-pinned-1.c: Adjust for new behaviour.
* testsuite/libgomp.c/alloc-pinned-2.c: Likewise.
* testsuite/libgomp.c/alloc-pinned-5.c: Likewise.
* testsuite/libgomp.c/alloc-pinned-8.c: New test.
* usmpin-allocator.c: New file.
---
libgomp/Makefile.am | 2 +-
libgomp/Makefile.in | 7 +-
libgomp/config/linux/allocator.c | 91 ++++--
libgomp/libgomp.h | 10 +
libgomp/testsuite/libgomp.c/alloc-pinned-8.c | 127 ++++++++
libgomp/usmpin-allocator.c | 319 +++++++++++++++++++
6 files changed, 523 insertions(+), 33 deletions(-)
create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-8.c
create mode 100644 libgomp/usmpin-allocator.c
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v3-0006-libgomp-fine-grained-pinned-memory-allocator.patch --]
[-- Type: text/x-patch; name="v3-0006-libgomp-fine-grained-pinned-memory-allocator.patch", Size: 20011 bytes --]
diff --git a/libgomp/Makefile.am b/libgomp/Makefile.am
index 1871590596d..9d41ed886d1 100644
--- a/libgomp/Makefile.am
+++ b/libgomp/Makefile.am
@@ -72,7 +72,7 @@ libgomp_la_SOURCES = alloc.c atomic.c barrier.c critical.c env.c error.c \
target.c splay-tree.c libgomp-plugin.c oacc-parallel.c oacc-host.c \
oacc-init.c oacc-mem.c oacc-async.c oacc-plugin.c oacc-cuda.c \
priority_queue.c affinity-fmt.c teams.c allocator.c oacc-profiling.c \
- oacc-target.c target-indirect.c
+ oacc-target.c target-indirect.c usmpin-allocator.c
include $(top_srcdir)/plugin/Makefrag.am
diff --git a/libgomp/Makefile.in b/libgomp/Makefile.in
index 56a6beab867..96fa9faf6a4 100644
--- a/libgomp/Makefile.in
+++ b/libgomp/Makefile.in
@@ -219,7 +219,8 @@ am_libgomp_la_OBJECTS = alloc.lo atomic.lo barrier.lo critical.lo \
oacc-parallel.lo oacc-host.lo oacc-init.lo oacc-mem.lo \
oacc-async.lo oacc-plugin.lo oacc-cuda.lo priority_queue.lo \
affinity-fmt.lo teams.lo allocator.lo oacc-profiling.lo \
- oacc-target.lo target-indirect.lo $(am__objects_1)
+ oacc-target.lo target-indirect.lo usmpin-allocator.lo \
+ $(am__objects_1)
libgomp_la_OBJECTS = $(am_libgomp_la_OBJECTS)
AM_V_P = $(am__v_P_@AM_V@)
am__v_P_ = $(am__v_P_@AM_DEFAULT_V@)
@@ -552,7 +553,8 @@ libgomp_la_SOURCES = alloc.c atomic.c barrier.c critical.c env.c \
oacc-parallel.c oacc-host.c oacc-init.c oacc-mem.c \
oacc-async.c oacc-plugin.c oacc-cuda.c priority_queue.c \
affinity-fmt.c teams.c allocator.c oacc-profiling.c \
- oacc-target.c target-indirect.c $(am__append_3)
+ oacc-target.c target-indirect.c usmpin-allocator.c \
+ $(am__append_3)
# Nvidia PTX OpenACC plugin.
@PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_version_info = -version-info $(libtool_VERSION)
@@ -786,6 +788,7 @@ distclean-compile:
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/team.Plo@am__quote@
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/teams.Plo@am__quote@
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/time.Plo@am__quote@
+@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/usmpin-allocator.Plo@am__quote@
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/work.Plo@am__quote@
.c.o:
diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c
index 8d681b5ec50..de53ca366a6 100644
--- a/libgomp/config/linux/allocator.c
+++ b/libgomp/config/linux/allocator.c
@@ -53,6 +53,7 @@
#define _GNU_SOURCE
#include <sys/mman.h>
+#include <unistd.h>
#include <string.h>
#include <assert.h>
#include "libgomp.h"
@@ -74,6 +75,16 @@ GOMP_enable_pinned_mode ()
static int using_device_for_page_locked
= /* uninitialized */ -1;
+
+static usmpin_ctx_p pin_ctx = NULL;
+static pthread_once_t ctxlock = PTHREAD_ONCE_INIT;
+
+static void
+linux_init_pin_ctx ()
+{
+ pin_ctx = usmpin_init_context ();
+}
+
static void *
linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin,
bool init0)
@@ -82,7 +93,7 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin,
__FUNCTION__, (unsigned long long) memspace,
(unsigned long long) size, pin, init0);
- void *addr;
+ void *addr = NULL;
/* Explicit pinning may not be required. */
pin = pin && !always_pinned_mode;
@@ -108,23 +119,44 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin,
}
if (using_device == 0)
{
- gomp_debug (0, " mmap\n");
- addr = mmap (NULL, size, PROT_READ | PROT_WRITE,
- MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
- if (addr == MAP_FAILED)
- addr = NULL;
- else
- {
- /* 'mmap' zero-initializes. */
- init0 = false;
+ static int pagesize = 0;
+ static void *addrhint = NULL;
- gomp_debug (0, " mlock\n");
- if (mlock (addr, size))
+ if (!pagesize)
+ pagesize = sysconf(_SC_PAGE_SIZE);
+
+ while (1)
+ {
+ addr = usmpin_alloc (pin_ctx, size);
+ if (addr)
+ break;
+
+ gomp_debug (0, " mmap\n");
+
+ /* Round up to a whole page. */
+ size_t misalignment = size % pagesize;
+ size_t mmap_size = (misalignment > 0
+ ? size + pagesize - misalignment
+ : size);
+ void *newpage = mmap (addrhint, mmap_size, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (newpage == MAP_FAILED)
+ break;
+ else
{
- gomp_debug (0, "libgomp: failed to pin %ld bytes of"
- " memory (ulimit too low?)\n", size);
- munmap (addr, size);
- addr = NULL;
+ gomp_debug (0, " mlock\n");
+ if (mlock (newpage, size))
+ {
+ gomp_debug (0, "libgomp: failed to pin %ld bytes of"
+ " memory (ulimit too low?)\n", size);
+ munmap (newpage, size);
+ break;
+ }
+
+ addrhint = newpage + mmap_size;
+
+ pthread_once (&ctxlock, linux_init_pin_ctx);
+ usmpin_register_memory (pin_ctx, newpage, mmap_size);
}
}
}
@@ -176,8 +208,7 @@ linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size,
if (using_device == 1)
gomp_page_locked_host_free (addr);
else
- /* 'munlock'ing is implicit with following 'munmap'. */
- munmap (addr, size);
+ usmpin_free (pin_ctx, addr);
}
else
free (addr);
@@ -195,29 +226,29 @@ linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr,
if (oldpin && pin)
{
- /* We can only expect to be able to just 'mremap' if not using a device
- for page-locked memory. */
int using_device
= __atomic_load_n (&using_device_for_page_locked,
MEMMODEL_RELAXED);
gomp_debug (0, " using_device=%d\n",
using_device);
- if (using_device != 0)
- goto manual_realloc;
-
- gomp_debug (0, " mremap\n");
- void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE);
- if (newaddr == MAP_FAILED)
- return NULL;
- return newaddr;
+ /* The device plugin API does not support realloc,
+ but the usmpin allocator does. */
+ if (using_device == 0)
+ {
+ /* This can fail if there is insufficient pinned memory free. */
+ void *newaddr = usmpin_realloc (pin_ctx, addr, size);
+ if (newaddr)
+ return newaddr;
+ }
}
else if (oldpin || pin)
- goto manual_realloc;
+ /* Moving from pinned to unpinned memory cannot be done in-place. */
+ ;
else
return realloc (addr, size);
-manual_realloc:;
+ /* In-place reallocation failed. Fall back to copy. */
void *newaddr = linux_memspace_alloc (memspace, size, pin, false);
if (newaddr)
{
diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h
index 8c9e50d88fd..8f56ac01f3a 100644
--- a/libgomp/libgomp.h
+++ b/libgomp/libgomp.h
@@ -1655,4 +1655,14 @@ gomp_thread_to_pthread_t (struct gomp_thread *thr)
}
#endif
+/* usmpin-allocator.c */
+
+typedef struct usmpin_context *usmpin_ctx_p;
+
+usmpin_ctx_p usmpin_init_context ();
+void usmpin_register_memory (usmpin_ctx_p ctx, char *base, size_t size);
+void *usmpin_alloc (usmpin_ctx_p ctx, size_t size);
+void usmpin_free (usmpin_ctx_p ctx, void *addr);
+void *usmpin_realloc (usmpin_ctx_p ctx, void *addr, size_t newsize);
+
#endif /* LIBGOMP_H */
diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-8.c b/libgomp/testsuite/libgomp.c/alloc-pinned-8.c
new file mode 100644
index 00000000000..fc5986805c7
--- /dev/null
+++ b/libgomp/testsuite/libgomp.c/alloc-pinned-8.c
@@ -0,0 +1,127 @@
+/* { dg-do run } */
+
+/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
+
+/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */
+
+/* Test that pinned memory works for small allocations. */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#ifdef __linux__
+#include <sys/types.h>
+#include <unistd.h>
+
+#include <sys/mman.h>
+#include <sys/resource.h>
+
+#define PAGE_SIZE sysconf(_SC_PAGESIZE)
+#define CHECK_SIZE(SIZE) { \
+ struct rlimit limit; \
+ if (getrlimit (RLIMIT_MEMLOCK, &limit) \
+ || limit.rlim_cur <= SIZE) \
+ fprintf (stderr, "unsufficient lockable memory; please increase ulimit\n"); \
+ }
+
+int
+get_pinned_mem ()
+{
+ int pid = getpid ();
+ char buf[100];
+ sprintf (buf, "/proc/%d/status", pid);
+
+ FILE *proc = fopen (buf, "r");
+ if (!proc)
+ abort ();
+ while (fgets (buf, 100, proc))
+ {
+ int val;
+ if (sscanf (buf, "VmLck: %d", &val))
+ {
+ fclose (proc);
+ return val;
+ }
+ }
+ abort ();
+}
+#else
+#define PAGE_SIZE 1 /* unknown */
+#define CHECK_SIZE(SIZE) fprintf (stderr, "OS unsupported\n");
+
+int
+get_pinned_mem ()
+{
+ return 0;
+}
+#endif
+
+static void
+verify0 (char *p, size_t s)
+{
+ for (size_t i = 0; i < s; ++i)
+ if (p[i] != 0)
+ abort ();
+}
+
+#include <omp.h>
+
+int
+main ()
+{
+ /* Choose a small size where all our allocations fit on one page. */
+ const int SIZE = 10;
+ CHECK_SIZE (SIZE*4);
+
+ const omp_alloctrait_t traits[] = {
+ { omp_atk_pinned, 1 }
+ };
+ omp_allocator_handle_t allocator = omp_init_allocator (omp_default_mem_space, 1, traits);
+
+ // Sanity check
+ if (get_pinned_mem () != 0)
+ abort ();
+
+ void *p = omp_alloc (SIZE, allocator);
+ if (!p)
+ abort ();
+
+ int amount = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount != 0)
+ abort ();
+#else
+ if (amount == 0)
+ abort ();
+#endif
+
+ p = omp_realloc (p, SIZE * 2, allocator, allocator);
+
+ int amount2 = get_pinned_mem ();
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (amount2 != 0)
+ abort ();
+#else
+ /* A small allocation should not allocate another page. */
+ if (amount2 != amount)
+ abort ();
+#endif
+
+ p = omp_calloc (1, SIZE, allocator);
+
+#ifdef OFFLOAD_DEVICE_NVPTX
+ /* This doesn't show up as process 'VmLck'ed memory. */
+ if (get_pinned_mem () != 0)
+ abort ();
+#else
+ /* A small allocation should not allocate another page. */
+ if (get_pinned_mem () != amount2)
+ abort ();
+#endif
+
+ verify0 (p, SIZE);
+
+ return 0;
+}
diff --git a/libgomp/usmpin-allocator.c b/libgomp/usmpin-allocator.c
new file mode 100644
index 00000000000..311bda5054e
--- /dev/null
+++ b/libgomp/usmpin-allocator.c
@@ -0,0 +1,319 @@
+/* Copyright (C) 2023 Free Software Foundation, Inc.
+
+ This file is part of the GNU Offloading and Multi Processing Library
+ (libgomp).
+
+ Libgomp is free software; you can redistribute it and/or modify it
+ under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 3, or (at your option)
+ any later version.
+
+ Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY
+ WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+ FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ more details.
+
+ Under Section 7 of GPL version 3, you are granted additional
+ permissions described in the GCC Runtime Library Exception, version
+ 3.1, as published by the Free Software Foundation.
+
+ You should have received a copy of the GNU General Public License and
+ a copy of the GCC Runtime Library Exception along with this program;
+ see the files COPYING3 and COPYING.RUNTIME respectively. If not, see
+ <http://www.gnu.org/licenses/>. */
+
+/* This is a simple "malloc" implementation intended for use with Unified
+ Shared Memory and Pinned Memory. It allocates memory from a pool allocated
+ and configured by the device plugin (for USM), or the OS-specific allocator
+ (for pinned).
+
+ This implementation keeps the allocated/free chain in a side-table (splay
+ tree) to ensure that the allocation routine does not migrate all the USM
+ pages back into host memory. Keeping the meta-data elsewhere is also useful
+ for pinned memory, which is typically an extremely limited resource. */
+
+#include <string.h>
+#include "libgomp.h"
+
+/* Use a splay tree to track allocations. */
+
+typedef struct usmpin_splay_tree_node_s *usmpin_splay_tree_node;
+typedef struct usmpin_splay_tree_s *usmpin_splay_tree;
+typedef struct usmpin_splay_tree_key_s *usmpin_splay_tree_key;
+
+struct usmpin_splay_tree_key_s {
+ void *base;
+ size_t size;
+};
+
+static inline int
+usmpin_splay_compare (usmpin_splay_tree_key x, usmpin_splay_tree_key y)
+{
+ return (x->base == y->base ? 0
+ : x->base > y->base ? 1
+ : -1);
+}
+
+#define splay_tree_prefix usmpin
+#include "splay-tree.h"
+
+/* 128-byte granularity means GPU cache-line aligned. */
+#define ALIGN(VAR) (((VAR) + 127) & ~127)
+
+/* The context data prevents the need for global state. */
+struct usmpin_context {
+ int lock;
+ struct usmpin_splay_tree_s allocations;
+ struct usmpin_splay_tree_s free_space;
+};
+
+usmpin_ctx_p
+usmpin_init_context ()
+{
+ return calloc (1, sizeof (struct usmpin_context));
+}
+
+/* Coalesce contiguous free space into one entry. This considers the entries
+ either side of the root node only, so it should be called each time a new
+ entry in inserted into the root. */
+
+static void
+usmpin_coalesce_free_space (usmpin_ctx_p ctx)
+{
+ usmpin_splay_tree_node prev, next, node = ctx->free_space.root;
+
+ for (prev = node->left; prev && prev->right; prev = prev->right)
+ ;
+ for (next = node->right; next && next->left; next = next->left)
+ ;
+
+ /* Coalesce adjacent free chunks. */
+ if (next
+ && node->key.base + node->key.size == next->key.base)
+ {
+ /* Free chunk follows. */
+ node->key.size += next->key.size;
+ usmpin_splay_tree_remove (&ctx->free_space, &next->key);
+ free (next);
+ }
+ if (prev
+ && prev->key.base + prev->key.size == node->key.base)
+ {
+ /* Free chunk precedes. */
+ prev->key.size += node->key.size;
+ usmpin_splay_tree_remove (&ctx->free_space, &node->key);
+ free (node);
+ }
+}
+
+/* Add a new memory region into the free chain. This is how the USM heap is
+ initialized and extended. If the new region is contiguous with an existing
+ region then any free space will be coalesced. */
+
+void
+usmpin_register_memory (usmpin_ctx_p ctx, char *base, size_t size)
+{
+ if (base == NULL || ctx == NULL)
+ return;
+
+ while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1)
+ ;
+
+ usmpin_splay_tree_node node;
+ node = malloc (sizeof (struct usmpin_splay_tree_node_s));
+ node->key.base = base;
+ node->key.size = size;
+ node->left = NULL;
+ node->right = NULL;
+ usmpin_splay_tree_insert (&ctx->free_space, node);
+ usmpin_coalesce_free_space (ctx);
+
+ __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE);
+}
+
+/* This splay_tree_foreach callback selects the first free space large enough
+ to hold the allocation needed. Since the splay_tree walk may start in the
+ middle the "first" isn't necessarily the "leftmost" entry. */
+
+struct usmpin_callback_data {
+ size_t size;
+ usmpin_splay_tree_node found;
+};
+
+static int
+usmpin_alloc_callback (usmpin_splay_tree_key key, void *data)
+{
+ struct usmpin_callback_data *cbd = (struct usmpin_callback_data *)data;
+
+ if (key->size >= cbd->size)
+ {
+ cbd->found = (usmpin_splay_tree_node)key;
+ return 1;
+ }
+
+ return 0;
+}
+
+/* USM "malloc". Selects and moves and address range from ctx->free_space to
+ ctx->allocations, while leaving any excess in ctx->free_space. */
+
+void *
+usmpin_alloc (usmpin_ctx_p ctx, size_t size)
+{
+ if (ctx == NULL)
+ return NULL;
+
+ /* Memory is allocated in N-byte granularity. */
+ size = ALIGN (size);
+
+ /* Acquire the lock. */
+ while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1)
+ ;
+
+ if (!ctx->free_space.root)
+ {
+ /* No memory registered, or no free space. */
+ __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE);
+ return NULL;
+ }
+
+ /* Find a suitable free block. */
+ struct usmpin_callback_data cbd = {size, NULL};
+ usmpin_splay_tree_foreach_lazy (&ctx->free_space, usmpin_alloc_callback,
+ &cbd);
+ usmpin_splay_tree_node freenode = cbd.found;
+
+ void *result = NULL;
+ if (freenode)
+ {
+ /* Allocation successful. */
+ result = freenode->key.base;
+ usmpin_splay_tree_node allocnode = malloc (sizeof (*allocnode));
+ allocnode->key.base = result;
+ allocnode->key.size = size;
+ allocnode->left = NULL;
+ allocnode->right = NULL;
+ usmpin_splay_tree_insert (&ctx->allocations, allocnode);
+
+ /* Update the free chain. */
+ size_t stillfree_size = freenode->key.size - size;
+ if (stillfree_size > 0)
+ {
+ freenode->key.base = freenode->key.base + size;
+ freenode->key.size = stillfree_size;
+ }
+ else
+ {
+ usmpin_splay_tree_remove (&ctx->free_space, &freenode->key);
+ free (freenode);
+ }
+ }
+
+ /* Release the lock. */
+ __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE);
+
+ return result;
+}
+
+/* USM "free". Moves an address range from ctx->allocations to
+ ctx->free_space and merges that record with any contiguous free memory. */
+
+void
+usmpin_free (usmpin_ctx_p ctx, void *addr)
+{
+ if (ctx == NULL)
+ return;
+
+ /* Acquire the lock. */
+ while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1)
+ ;
+
+ /* Convert the memory map to free. */
+ struct usmpin_splay_tree_key_s key = {addr};
+ usmpin_splay_tree_key found = usmpin_splay_tree_lookup (&ctx->allocations,
+ &key);
+ if (!found)
+ GOMP_PLUGIN_fatal ("invalid free");
+ usmpin_splay_tree_remove (&ctx->allocations, &key);
+ usmpin_splay_tree_insert (&ctx->free_space, (usmpin_splay_tree_node)found);
+ usmpin_coalesce_free_space (ctx);
+
+ /* Release the lock. */
+ __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE);
+}
+
+/* USM "realloc". Works in-place, if possible; reallocates otherwise. */
+
+void *
+usmpin_realloc (usmpin_ctx_p ctx, void *addr, size_t newsize)
+{
+ if (ctx == NULL)
+ return NULL;
+
+ newsize = ALIGN (newsize);
+
+ /* Acquire the lock. */
+ while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1)
+ ;
+
+ /* Convert the memory map to free. */
+ struct usmpin_splay_tree_key_s key = {addr};
+ usmpin_splay_tree_key found = usmpin_splay_tree_lookup (&ctx->allocations,
+ &key);
+ if (!found)
+ GOMP_PLUGIN_fatal ("invalid realloc");
+
+ if (newsize == found->size)
+ ; /* Nothing to do. */
+ else if (newsize < found->size)
+ {
+ /* We're reducing the allocation size. */
+ usmpin_splay_tree_node newfree = malloc (sizeof (*newfree));
+ newfree->key.base = found->base + newsize;
+ newfree->key.size = found->size - newsize;
+ newfree->left = NULL;
+ newfree->right = NULL;
+ usmpin_splay_tree_insert (&ctx->free_space, newfree);
+ usmpin_coalesce_free_space (ctx);
+ }
+ else
+ {
+ /* We're extending the allocation. */
+ struct usmpin_splay_tree_key_s freekey = {addr + found->size};
+ usmpin_splay_tree_key foundfree;
+ foundfree = usmpin_splay_tree_lookup (&ctx->free_space, &freekey);
+ if (foundfree && foundfree->size >= newsize - found->size)
+ {
+ /* Allocation can be expanded in place. */
+ foundfree->base += found->size;
+ foundfree->size -= newsize - found->size;
+ found->size = newsize;
+
+ if (foundfree->size == 0)
+ usmpin_splay_tree_remove (&ctx->free_space, &freekey);
+ }
+ else
+ {
+ /* Allocation must be relocated.
+ Release the lock and use alloc/free. */
+ __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE);
+
+ void *newaddr = usmpin_alloc (ctx, newsize);
+ if (!newaddr)
+ return NULL;
+
+ memcpy (newaddr, addr, found->size);
+ usmpin_free (ctx, addr);
+ return newaddr;
+ }
+ }
+
+ /* Release the lock. */
+ __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE);
+ return addr;
+}
+
+/* Include the splay tree code inline, with the prefixes added. */
+#define splay_tree_prefix usmpin
+#define splay_tree_c
+#include "splay-tree.h"
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3 1/6] libgomp: basic pinned memory on Linux
2023-12-11 17:04 ` [PATCH v3 1/6] libgomp: basic pinned memory on Linux Andrew Stubbs
@ 2023-12-12 9:02 ` Tobias Burnus
2023-12-13 14:28 ` Andrew Stubbs
0 siblings, 1 reply; 14+ messages in thread
From: Tobias Burnus @ 2023-12-12 9:02 UTC (permalink / raw)
To: Andrew Stubbs, gcc-patches
On 11.12.23 18:04, Andrew Stubbs wrote:
> Implement the OpenMP pinned memory trait on Linux hosts using the mlock
> syscall. Pinned allocations are performed using mmap, not malloc, to ensure
> that they can be unpinned safely when freed.
>
> This implementation will work OK for page-scale allocations, and finer-grained
> allocations will be implemented in a future patch.
LGTM.
Thanks,
Tobias
> libgomp/ChangeLog:
>
> * allocator.c (MEMSPACE_ALLOC): Add PIN.
> (MEMSPACE_CALLOC): Add PIN.
> (MEMSPACE_REALLOC): Add PIN.
> (MEMSPACE_FREE): Add PIN.
> (MEMSPACE_VALIDATE): Add PIN.
> (omp_init_allocator): Use MEMSPACE_VALIDATE to check pinning.
> (omp_aligned_alloc): Add pinning to all MEMSPACE_* calls.
> (omp_aligned_calloc): Likewise.
> (omp_realloc): Likewise.
> (omp_free): Likewise.
> * config/linux/allocator.c: New file.
> * config/nvptx/allocator.c (MEMSPACE_ALLOC): Add PIN.
> (MEMSPACE_CALLOC): Add PIN.
> (MEMSPACE_REALLOC): Add PIN.
> (MEMSPACE_FREE): Add PIN.
> (MEMSPACE_VALIDATE): Add PIN.
> * config/gcn/allocator.c (MEMSPACE_ALLOC): Add PIN.
> (MEMSPACE_CALLOC): Add PIN.
> (MEMSPACE_REALLOC): Add PIN.
> (MEMSPACE_FREE): Add PIN.
> * libgomp.texi: Switch pinned trait to supported.
> (MEMSPACE_VALIDATE): Add PIN.
> * testsuite/libgomp.c/alloc-pinned-1.c: New test.
> * testsuite/libgomp.c/alloc-pinned-2.c: New test.
> * testsuite/libgomp.c/alloc-pinned-3.c: New test.
> * testsuite/libgomp.c/alloc-pinned-4.c: New test.
>
> Co-Authored-By: Thomas Schwinge <thomas@codesourcery.com>
> ---
> libgomp/allocator.c | 65 +++++---
> libgomp/config/gcn/allocator.c | 21 +--
> libgomp/config/linux/allocator.c | 111 +++++++++++++
> libgomp/config/nvptx/allocator.c | 21 +--
> libgomp/libgomp.texi | 3 +-
> libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 115 ++++++++++++++
> libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 120 ++++++++++++++
> libgomp/testsuite/libgomp.c/alloc-pinned-3.c | 156 +++++++++++++++++++
> libgomp/testsuite/libgomp.c/alloc-pinned-4.c | 150 ++++++++++++++++++
> 9 files changed, 716 insertions(+), 46 deletions(-)
> create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-1.c
> create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-2.c
> create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-3.c
> create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-4.c
>
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3 5/6] libgomp, nvptx: Cuda pinned memory
2023-12-11 17:04 ` [PATCH v3 5/6] libgomp, nvptx: Cuda pinned memory Andrew Stubbs
@ 2023-12-12 9:10 ` Tobias Burnus
0 siblings, 0 replies; 14+ messages in thread
From: Tobias Burnus @ 2023-12-12 9:10 UTC (permalink / raw)
To: Andrew Stubbs, gcc-patches
On 11.12.23 18:04, Andrew Stubbs wrote:
> Use Cuda to pin memory, instead of Linux mlock, when available.
>
> There are two advantages: firstly, this gives a significant speed boost for
> NVPTX offloading, and secondly, it side-steps the usual OS ulimit/rlimit
> setting.
>
> The design adds a device independent plugin API for allocating pinned memory,
> and then implements it for NVPTX. At present, the other supported devices do
> not have equivalent capabilities (or requirements).
LGTM. If you temporarily back out .texi's "@code{ompx_pinned_mem_alloc}
allocator or" it can go in immediately after 1/6, i.e. before the
ompx_pinned_mem_alloc patch.
As discussed in previous thread for the previous patch, I think the init
should be done in the worker function of gomp_init_targets_once().
As discussed elsewhere, the patch can go in as is and I will provide a
follow-up patch.
Thanks,
Tobias
> libgomp/ChangeLog:
>
> * config/linux/allocator.c: Include assert.h.
> (using_device_for_page_locked): New variable.
> (linux_memspace_alloc): Add init0 parameter. Support device pinning.
> (linux_memspace_calloc): Set init0 to true.
> (linux_memspace_free): Support device pinning.
> (linux_memspace_realloc): Support device pinning.
> (MEMSPACE_ALLOC): Set init0 to false.
> * libgomp-plugin.h
> (GOMP_OFFLOAD_page_locked_host_alloc): New prototype.
> (GOMP_OFFLOAD_page_locked_host_free): Likewise.
> * libgomp.h (gomp_page_locked_host_alloc): Likewise.
> (gomp_page_locked_host_free): Likewise.
> (struct gomp_device_descr): Add page_locked_host_alloc_func and
> page_locked_host_free_func.
> * libgomp.texi: Adjust the docs for the pinned trait.
> * libgomp_g.h (GOMP_enable_pinned_mode): New prototype.
> * plugin/plugin-nvptx.c
> (GOMP_OFFLOAD_page_locked_host_alloc): New function.
> (GOMP_OFFLOAD_page_locked_host_free): Likewise.
> * target.c (device_for_page_locked): New variable.
> (get_device_for_page_locked): New function.
> (gomp_page_locked_host_alloc): Likewise.
> (gomp_page_locked_host_free): Likewise.
> (gomp_load_plugin_for_device): Add page_locked_host_alloc and
> page_locked_host_free.
> * testsuite/libgomp.c/alloc-pinned-1.c: Change expectations for NVPTX
> devices.
> * testsuite/libgomp.c/alloc-pinned-2.c: Likewise.
> * testsuite/libgomp.c/alloc-pinned-3.c: Likewise.
> * testsuite/libgomp.c/alloc-pinned-4.c: Likewise.
> * testsuite/libgomp.c/alloc-pinned-5.c: Likewise.
> * testsuite/libgomp.c/alloc-pinned-6.c: Likewise.
>
> Co-Authored-By: Thomas Schwinge <thomas@codesourcery.com>
> ---
> libgomp/config/linux/allocator.c | 137 ++++++++++++++-----
> libgomp/libgomp-plugin.h | 2 +
> libgomp/libgomp.h | 4 +
> libgomp/libgomp.texi | 11 +-
> libgomp/libgomp_g.h | 1 +
> libgomp/plugin/plugin-nvptx.c | 42 ++++++
> libgomp/target.c | 136 ++++++++++++++++++
> libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 26 ++++
> libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 26 ++++
> libgomp/testsuite/libgomp.c/alloc-pinned-3.c | 45 +++++-
> libgomp/testsuite/libgomp.c/alloc-pinned-4.c | 44 +++++-
> libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 26 ++++
> libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 35 ++++-
> 13 files changed, 487 insertions(+), 48 deletions(-)
>
--
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc
2023-12-11 17:04 ` [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc Andrew Stubbs
@ 2023-12-12 10:05 ` Tobias Burnus
2023-12-12 16:17 ` Andrew Stubbs
0 siblings, 1 reply; 14+ messages in thread
From: Tobias Burnus @ 2023-12-12 10:05 UTC (permalink / raw)
To: gcc-patches, Andrew Stubbs
Hi Andrew,
On 11.12.23 18:04, Andrew Stubbs wrote:
> This creates a new predefined allocator as a shortcut for using pinned
> memory with OpenMP. The name uses the OpenMP extension space and is
> intended to be consistent with other OpenMP implementations currently in
> development.
Discussed this with Jakub - and 9 does not permit to have a contiguous
range of numbers if OpenMP ever extends this,
Thus, maybe start the ompx_ with 100.
We were also pondering whether it should be ompx_gnu_pinned_mem_alloc or
ompx_pinned_mem_alloc.
The only other compiler supporting this flag seems to be IBM; their
compiler uses ompx_pinned_mem_alloc with the same meaning:
https://www.ibm.com/support/pages/system/files/inline-files/OMP5_User_Reference.pdf
(page 5)
As the obvious meaning is what both compilers have, I am fine without
the "gnu" infix, which Jakub accepted.
* * *
And you have not updated the compiler itself to support more this new
allocator. Cf.
https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/c-c++-common/gomp/allocate-9.c#L23-L28
That file gives an overview what needs to be changed:
* The check functions mentioned there (seemingly for two ranges now)
* Update the OMP_ALLOCATOR env var parser in env.c
* That linked testcase (and possibly some some more) should be updated,
also to ensure that the new allocator is accepted + to check for new
unsupported values (99, 101 ?)
If we now leave gaps, the const_assert in libgomp/allocator.c probably
needs to be updated as well.
* * *
Glancing through the patches, for test cases, I think you should
'abort()' in CHECK_SIZE if it fails (rlimit issue or not supported
system). Or do you think that the results are still could make sense
when continuing and possibly failing later?
Tobias
> The allocator is equivalent to using a custom allocator with the pinned
> trait and the null fallback trait.
>
> libgomp/ChangeLog:
>
> * allocator.c (omp_max_predefined_alloc): Update.
> (predefined_alloc_mapping): Add ompx_pinned_mem_alloc entry.
> (omp_aligned_alloc): Support ompx_pinned_mem_alloc.
> (omp_free): Likewise.
> (omp_aligned_calloc): Likewise.
> (omp_realloc): Likewise.
> * libgomp.texi: Document ompx_pinned_mem_alloc.
> * omp.h.in (omp_allocator_handle_t): Add ompx_pinned_mem_alloc.
> * omp_lib.f90.in: Add ompx_pinned_mem_alloc.
> * testsuite/libgomp.c/alloc-pinned-5.c: New test.
> * testsuite/libgomp.c/alloc-pinned-6.c: New test.
> * testsuite/libgomp.fortran/alloc-pinned-1.f90: New test.
>
> Co-Authored-By: Thomas Schwinge<thomas@codesourcery.com>
> ---
> libgomp/allocator.c | 58 ++++++----
> libgomp/libgomp.texi | 7 +-
> libgomp/omp.h.in | 1 +
> libgomp/omp_lib.f90.in | 2 +
> libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 103 ++++++++++++++++++
> libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 101 +++++++++++++++++
> .../libgomp.fortran/alloc-pinned-1.f90 | 16 +++
> 7 files changed, 268 insertions(+), 20 deletions(-)
> create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-5.c
> create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-6.c
> create mode 100644 libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90
>
>
> v3-0002-libgomp-openmp-Add-ompx_pinned_mem_alloc.patch
>
> diff --git a/libgomp/allocator.c b/libgomp/allocator.c
> index 666adf9a3a9..6c69c4f008f 100644
> --- a/libgomp/allocator.c
> +++ b/libgomp/allocator.c
> @@ -35,7 +35,7 @@
> #include <dlfcn.h>
> #endif
>
> -#define omp_max_predefined_alloc omp_thread_mem_alloc
> +#define omp_max_predefined_alloc ompx_pinned_mem_alloc
>
> /* These macros may be overridden in config/<target>/allocator.c.
> The defaults (no override) are to return NULL for pinned memory requests
> @@ -78,6 +78,7 @@ static const omp_memspace_handle_t predefined_alloc_mapping[] = {
> omp_low_lat_mem_space, /* omp_cgroup_mem_alloc (implementation defined). */
> omp_low_lat_mem_space, /* omp_pteam_mem_alloc (implementation defined). */
> omp_low_lat_mem_space, /* omp_thread_mem_alloc (implementation defined). */
> + omp_default_mem_space, /* ompx_pinned_mem_alloc. */
> };
>
> #define ARRAY_SIZE(A) (sizeof (A) / sizeof ((A)[0]))
> @@ -623,8 +624,10 @@ retry:
> memspace = (allocator_data
> ? allocator_data->memspace
> : predefined_alloc_mapping[allocator]);
> - ptr = MEMSPACE_ALLOC (memspace, new_size,
> - allocator_data && allocator_data->pinned);
> + int pinned = (allocator_data
> + ? allocator_data->pinned
> + : allocator == ompx_pinned_mem_alloc);
> + ptr = MEMSPACE_ALLOC (memspace, new_size, pinned);
> }
> if (ptr == NULL)
> goto fail;
> @@ -645,7 +648,8 @@ retry:
> fail:;
> int fallback = (allocator_data
> ? allocator_data->fallback
> - : allocator == omp_default_mem_alloc
> + : (allocator == omp_default_mem_alloc
> + || allocator == ompx_pinned_mem_alloc)
> ? omp_atv_null_fb
> : omp_atv_default_mem_fb);
> switch (fallback)
> @@ -760,6 +764,7 @@ omp_free (void *ptr, omp_allocator_handle_t allocator)
> #endif
>
> memspace = predefined_alloc_mapping[data->allocator];
> + pinned = (data->allocator == ompx_pinned_mem_alloc);
> }
>
> MEMSPACE_FREE (memspace, data->ptr, data->size, pinned);
> @@ -933,8 +938,10 @@ retry:
> memspace = (allocator_data
> ? allocator_data->memspace
> : predefined_alloc_mapping[allocator]);
> - ptr = MEMSPACE_CALLOC (memspace, new_size,
> - allocator_data && allocator_data->pinned);
> + int pinned = (allocator_data
> + ? allocator_data->pinned
> + : allocator == ompx_pinned_mem_alloc);
> + ptr = MEMSPACE_CALLOC (memspace, new_size, pinned);
> }
> if (ptr == NULL)
> goto fail;
> @@ -955,7 +962,8 @@ retry:
> fail:;
> int fallback = (allocator_data
> ? allocator_data->fallback
> - : allocator == omp_default_mem_alloc
> + : (allocator == omp_default_mem_alloc
> + || allocator == ompx_pinned_mem_alloc)
> ? omp_atv_null_fb
> : omp_atv_default_mem_fb);
> switch (fallback)
> @@ -1165,11 +1173,14 @@ retry:
> else
> #endif
> if (prev_size)
> - new_ptr = MEMSPACE_REALLOC (allocator_data->memspace, data->ptr,
> - data->size, new_size,
> - (free_allocator_data
> - && free_allocator_data->pinned),
> - allocator_data->pinned);
> + {
> + int was_pinned = (free_allocator_data
> + ? free_allocator_data->pinned
> + : free_allocator == ompx_pinned_mem_alloc);
> + new_ptr = MEMSPACE_REALLOC (allocator_data->memspace, data->ptr,
> + data->size, new_size, was_pinned,
> + allocator_data->pinned);
> + }
> else
> new_ptr = MEMSPACE_ALLOC (allocator_data->memspace, new_size,
> allocator_data->pinned);
> @@ -1225,10 +1236,14 @@ retry:
> memspace = (allocator_data
> ? allocator_data->memspace
> : predefined_alloc_mapping[allocator]);
> + int was_pinned = (free_allocator_data
> + ? free_allocator_data->pinned
> + : free_allocator == ompx_pinned_mem_alloc);
> + int pinned = (allocator_data
> + ? allocator_data->pinned
> + : allocator == ompx_pinned_mem_alloc);
> new_ptr = MEMSPACE_REALLOC (memspace, data->ptr, data->size, new_size,
> - (free_allocator_data
> - && free_allocator_data->pinned),
> - allocator_data && allocator_data->pinned);
> + was_pinned, pinned);
> }
> if (new_ptr == NULL)
> goto fail;
> @@ -1262,8 +1277,10 @@ retry:
> memspace = (allocator_data
> ? allocator_data->memspace
> : predefined_alloc_mapping[allocator]);
> - new_ptr = MEMSPACE_ALLOC (memspace, new_size,
> - allocator_data && allocator_data->pinned);
> + int pinned = (allocator_data
> + ? allocator_data->pinned
> + : allocator == ompx_pinned_mem_alloc);
> + new_ptr = MEMSPACE_ALLOC (memspace, new_size, pinned);
> }
> if (new_ptr == NULL)
> goto fail;
> @@ -1318,7 +1335,9 @@ retry:
> was_memspace = (free_allocator_data
> ? free_allocator_data->memspace
> : predefined_alloc_mapping[free_allocator]);
> - int was_pinned = (free_allocator_data && free_allocator_data->pinned);
> + int was_pinned = (free_allocator_data
> + ? free_allocator_data->pinned
> + : free_allocator == ompx_pinned_mem_alloc);
> MEMSPACE_FREE (was_memspace, data->ptr, data->size, was_pinned);
> }
> return ret;
> @@ -1326,7 +1345,8 @@ retry:
> fail:;
> int fallback = (allocator_data
> ? allocator_data->fallback
> - : allocator == omp_default_mem_alloc
> + : (allocator == omp_default_mem_alloc
> + || allocator == ompx_pinned_mem_alloc)
> ? omp_atv_null_fb
> : omp_atv_default_mem_fb);
> switch (fallback)
> diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi
> index 5838311b7d8..1f73c49c9b8 100644
> --- a/libgomp/libgomp.texi
> +++ b/libgomp/libgomp.texi
> @@ -3015,6 +3015,7 @@ value.
> @item omp_cgroup_mem_alloc @tab omp_low_lat_mem_space (implementation defined)
> @item omp_pteam_mem_alloc @tab omp_low_lat_mem_space (implementation defined)
> @item omp_thread_mem_alloc @tab omp_low_lat_mem_space (implementation defined)
> +@item ompx_pinned_mem_alloc @tab omp_default_mem_space (GNU extension)
> @end multitable
>
> The predefined allocators use the default values for the traits,
> @@ -3040,7 +3041,7 @@ as listed below. Except that the last three allocators have the
> @item @code{fb_data} @tab @emph{unsupported as it needs an allocator handle}
> @tab (none)
> @item @code{pinned} @tab @code{true}, @code{false}
> - @tab @code{false}
> + @tab See below
> @item @code{partition} @tab @code{environment}, @code{nearest},
> @code{blocked}, @code{interleaved}
> @tab @code{environment}
> @@ -3051,6 +3052,10 @@ For the @code{fallback} trait, the default value is @code{null_fb} for the
> with device memory; for all other other allocators, it is @code{default_mem_fb}
> by default.
>
> +For the @code{pinned} trait, the default value is @code{true} for
> +predefined allocator @code{ompx_pinned_mem_alloc} (a GNU extension), and
> +@code{false} for all others.
> +
> Examples:
> @smallexample
> OMP_ALLOCATOR=omp_high_bw_mem_alloc
> diff --git a/libgomp/omp.h.in b/libgomp/omp.h.in
> index bd1286c2a3f..66989e693b7 100644
> --- a/libgomp/omp.h.in
> +++ b/libgomp/omp.h.in
> @@ -134,6 +134,7 @@ typedef enum omp_allocator_handle_t __GOMP_UINTPTR_T_ENUM
> omp_cgroup_mem_alloc = 6,
> omp_pteam_mem_alloc = 7,
> omp_thread_mem_alloc = 8,
> + ompx_pinned_mem_alloc = 9,
> __omp_allocator_handle_t_max__ = __UINTPTR_MAX__
> } omp_allocator_handle_t;
>
> diff --git a/libgomp/omp_lib.f90.in b/libgomp/omp_lib.f90.in
> index 755c0bf16ec..e27e4aacbdf 100644
> --- a/libgomp/omp_lib.f90.in
> +++ b/libgomp/omp_lib.f90.in
> @@ -158,6 +158,8 @@
> parameter :: omp_pteam_mem_alloc = 7
> integer (kind=omp_allocator_handle_kind), &
> parameter :: omp_thread_mem_alloc = 8
> + integer (kind=omp_allocator_handle_kind), &
> + parameter :: ompx_pinned_mem_alloc = 9
> integer (omp_memspace_handle_kind), &
> parameter :: omp_default_mem_space = 0
> integer (omp_memspace_handle_kind), &
> diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c
> new file mode 100644
> index 00000000000..18e6d20ca5b
> --- /dev/null
> +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c
> @@ -0,0 +1,103 @@
> +/* { dg-do run } */
> +
> +/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */
> +
> +/* Test that ompx_pinned_mem_alloc works. */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +
> +#ifdef __linux__
> +#include <sys/types.h>
> +#include <unistd.h>
> +
> +#include <sys/mman.h>
> +#include <sys/resource.h>
> +
> +#define PAGE_SIZE sysconf(_SC_PAGESIZE)
> +#define CHECK_SIZE(SIZE) { \
> + struct rlimit limit; \
> + if (getrlimit (RLIMIT_MEMLOCK, &limit) \
> + || limit.rlim_cur <= SIZE) \
> + fprintf (stderr, "unsufficient lockable memory; please increase ulimit\n"); \
> + }
> +
> +int
> +get_pinned_mem ()
> +{
> + int pid = getpid ();
> + char buf[100];
> + sprintf (buf, "/proc/%d/status", pid);
> +
> + FILE *proc = fopen (buf, "r");
> + if (!proc)
> + abort ();
> + while (fgets (buf, 100, proc))
> + {
> + int val;
> + if (sscanf (buf, "VmLck: %d", &val))
> + {
> + fclose (proc);
> + return val;
> + }
> + }
> + abort ();
> +}
> +#else
> +#define PAGE_SIZE 10000 * 1024 /* unknown */
> +#define CHECK_SIZE(SIZE) fprintf (stderr, "OS unsupported\n");
> +
> +int
> +get_pinned_mem ()
> +{
> + return 0;
> +}
> +#endif
> +
> +static void
> +verify0 (char *p, size_t s)
> +{
> + for (size_t i = 0; i < s; ++i)
> + if (p[i] != 0)
> + abort ();
> +}
> +
> +#include <omp.h>
> +
> +int
> +main ()
> +{
> + /* Allocate at least a page each time, allowing space for overhead,
> + but stay within the ulimit. */
> + const int SIZE = PAGE_SIZE - 128;
> + CHECK_SIZE (SIZE * 5);
> +
> + // Sanity check
> + if (get_pinned_mem () != 0)
> + abort ();
> +
> + void *p = omp_alloc (SIZE, ompx_pinned_mem_alloc);
> + if (!p)
> + abort ();
> +
> + int amount = get_pinned_mem ();
> + if (amount == 0)
> + abort ();
> +
> + p = omp_realloc (p, SIZE * 2, ompx_pinned_mem_alloc, ompx_pinned_mem_alloc);
> +
> + int amount2 = get_pinned_mem ();
> + if (amount2 <= amount)
> + abort ();
> +
> + /* SIZE*2 ensures that it doesn't slot into the space possibly
> + vacated by realloc. */
> + p = omp_calloc (1, SIZE * 2, ompx_pinned_mem_alloc);
> +
> + if (get_pinned_mem () <= amount2)
> + abort ();
> +
> + verify0 (p, SIZE * 2);
> +
> + return 0;
> +}
> diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c
> new file mode 100644
> index 00000000000..f80a0264f97
> --- /dev/null
> +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c
> @@ -0,0 +1,101 @@
> +/* { dg-do run } */
> +
> +/* Test that ompx_pinned_mem_alloc fails correctly. */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +
> +#ifdef __linux__
> +#include <sys/types.h>
> +#include <unistd.h>
> +
> +#include <sys/mman.h>
> +#include <sys/resource.h>
> +
> +#define PAGE_SIZE sysconf(_SC_PAGESIZE)
> +
> +int
> +get_pinned_mem ()
> +{
> + int pid = getpid ();
> + char buf[100];
> + sprintf (buf, "/proc/%d/status", pid);
> +
> + FILE *proc = fopen (buf, "r");
> + if (!proc)
> + abort ();
> + while (fgets (buf, 100, proc))
> + {
> + int val;
> + if (sscanf (buf, "VmLck: %d", &val))
> + {
> + fclose (proc);
> + return val;
> + }
> + }
> + abort ();
> +}
> +
> +void
> +set_pin_limit (int size)
> +{
> + struct rlimit limit;
> + if (getrlimit (RLIMIT_MEMLOCK, &limit))
> + abort ();
> + limit.rlim_cur = (limit.rlim_max < size ? limit.rlim_max : size);
> + if (setrlimit (RLIMIT_MEMLOCK, &limit))
> + abort ();
> +}
> +#else
> +#define PAGE_SIZE 10000 * 1024 /* unknown */
> +
> +int
> +get_pinned_mem ()
> +{
> + return 0;
> +}
> +
> +void
> +set_pin_limit ()
> +{
> +}
> +#endif
> +
> +#include <omp.h>
> +
> +int
> +main ()
> +{
> + /* Allocate at least a page each time, but stay within the ulimit. */
> + const int SIZE = PAGE_SIZE * 4;
> +
> + /* Ensure that the limit is smaller than the allocation. */
> + set_pin_limit (SIZE / 2);
> +
> + // Sanity check
> + if (get_pinned_mem () != 0)
> + abort ();
> +
> + // Should fail
> + void *p = omp_alloc (SIZE, ompx_pinned_mem_alloc);
> + if (p)
> + abort ();
> +
> + // Should fail
> + p = omp_calloc (1, SIZE, ompx_pinned_mem_alloc);
> + if (p)
> + abort ();
> +
> + // Should fail to realloc
> + void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc);
> + p = omp_realloc (notpinned, SIZE, ompx_pinned_mem_alloc, omp_default_mem_alloc);
> + if (!notpinned || p)
> + abort ();
> +
> + // No memory should have been pinned
> + int amount = get_pinned_mem ();
> + if (amount != 0)
> + abort ();
> +
> + return 0;
> +}
> diff --git a/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90 b/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90
> new file mode 100644
> index 00000000000..798dc3d5a12
> --- /dev/null
> +++ b/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90
> @@ -0,0 +1,16 @@
> +! Ensure that the ompx_pinned_mem_alloc predefined allocator is present and
> +! accepted. The majority of the functionality testing lives in the C tests.
> +!
> +! { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } }
> +
> +program main
> + use omp_lib
> + use ISO_C_Binding
> + implicit none (external, type)
> +
> + type(c_ptr) :: p
> +
> + p = omp_alloc (10_c_size_t, ompx_pinned_mem_alloc);
> + if (.not. c_associated (p)) stop 1
> + call omp_free (p, ompx_pinned_mem_alloc);
> +end program main
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc
2023-12-12 10:05 ` Tobias Burnus
@ 2023-12-12 16:17 ` Andrew Stubbs
2023-12-13 8:52 ` Jakub Jelinek
0 siblings, 1 reply; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-12 16:17 UTC (permalink / raw)
To: Tobias Burnus, gcc-patches
On 12/12/2023 10:05, Tobias Burnus wrote:
> Hi Andrew,
>
> On 11.12.23 18:04, Andrew Stubbs wrote:
>> This creates a new predefined allocator as a shortcut for using pinned
>> memory with OpenMP. The name uses the OpenMP extension space and is
>> intended to be consistent with other OpenMP implementations currently in
>> development.
>
> Discussed this with Jakub - and 9 does not permit to have a contiguous
> range of numbers if OpenMP ever extends this,
>
> Thus, maybe start the ompx_ with 100.
These numbers are not defined in any standard, are they? We can use
whatever enumeration we choose.
I'm happy to change them, but the *_mem_alloc numbers are used as an
index into a constant table to map them to the corresponding
*_mem_space, so do we really want to make it a sparse table?
> We were also pondering whether it should be ompx_gnu_pinned_mem_alloc or
> ompx_pinned_mem_alloc.
It's a long time ago now, and I'm struggling to remember, but I think
those names were agreed with some other parties (can't remember who
though, and I may be thinking of the ompx_unified_shared_mem_alloc that
is still to come).
> The only other compiler supporting this flag seems to be IBM; their
> compiler uses ompx_pinned_mem_alloc with the same meaning:
> https://www.ibm.com/support/pages/system/files/inline-files/OMP5_User_Reference.pdf
> (page 5)
>
> As the obvious meaning is what both compilers have, I am fine without
> the "gnu" infix, which Jakub accepted.
Good.
>
> * * *
>
> And you have not updated the compiler itself to support more this new
> allocator. Cf.
>
> https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/c-c++-common/gomp/allocate-9.c#L23-L28
>
> That file gives an overview what needs to be changed:
>
> * The check functions mentioned there (seemingly for two ranges now)
>
> * Update the OMP_ALLOCATOR env var parser in env.c
>
> * That linked testcase (and possibly some some more) should be updated,
> also to ensure that the new allocator is accepted + to check for new
> unsupported values (99, 101 ?)
>
> If we now leave gaps, the const_assert in libgomp/allocator.c probably
> needs to be updated as well.
>
> * * *
>
> Glancing through the patches, for test cases, I think you should
> 'abort()' in CHECK_SIZE if it fails (rlimit issue or not supported
> system). Or do you think that the results are still could make sense
> when continuing and possibly failing later?
Those were not meant to be part of the test, really, but rather to
demystify failures for future maintainers.
>
> Tobias
Thanks for the review.
Andrew
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc
2023-12-12 16:17 ` Andrew Stubbs
@ 2023-12-13 8:52 ` Jakub Jelinek
0 siblings, 0 replies; 14+ messages in thread
From: Jakub Jelinek @ 2023-12-13 8:52 UTC (permalink / raw)
To: Andrew Stubbs; +Cc: Tobias Burnus, gcc-patches
On Tue, Dec 12, 2023 at 04:17:47PM +0000, Andrew Stubbs wrote:
> On 12/12/2023 10:05, Tobias Burnus wrote:
> > Hi Andrew,
> >
> > On 11.12.23 18:04, Andrew Stubbs wrote:
> > > This creates a new predefined allocator as a shortcut for using pinned
> > > memory with OpenMP. The name uses the OpenMP extension space and is
> > > intended to be consistent with other OpenMP implementations currently in
> > > development.
> >
> > Discussed this with Jakub - and 9 does not permit to have a contiguous
> > range of numbers if OpenMP ever extends this,
> >
> > Thus, maybe start the ompx_ with 100.
>
> These numbers are not defined in any standard, are they?
omp_null_allocator is (it has to be zero), the others don't have value
specified in the OpenMP standard, but it is an ABI thing.
LLVM has
omp_null_allocator = 0,
omp_default_mem_alloc = 1,
omp_large_cap_mem_alloc = 2,
omp_const_mem_alloc = 3,
omp_high_bw_mem_alloc = 4,
omp_low_lat_mem_alloc = 5,
omp_cgroup_mem_alloc = 6,
omp_pteam_mem_alloc = 7,
omp_thread_mem_alloc = 8,
llvm_omp_target_host_mem_alloc = 100,
llvm_omp_target_shared_mem_alloc = 101,
llvm_omp_target_device_mem_alloc = 102,
KMP_ALLOCATOR_MAX_HANDLE = UINTPTR_MAX
and I think it is desirable to be ABI compatible here with the standard
values. So, I think it is better to have one range for the standard ones
(same values as other implementations) and another range for the extensions
(and given that LLVM uses 100-102 range, perhaps pick a different one, 200
or 150). And then 2 tables, one for the standard range, one for the
extensions.
Jakub
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v3 1/6] libgomp: basic pinned memory on Linux
2023-12-12 9:02 ` Tobias Burnus
@ 2023-12-13 14:28 ` Andrew Stubbs
[not found] ` <9ffed162-2197-4acd-b72c-1946dc899f86@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Andrew Stubbs @ 2023-12-13 14:28 UTC (permalink / raw)
To: Tobias Burnus, gcc-patches
On 12/12/2023 09:02, Tobias Burnus wrote:
> On 11.12.23 18:04, Andrew Stubbs wrote:
>> Implement the OpenMP pinned memory trait on Linux hosts using the mlock
>> syscall. Pinned allocations are performed using mmap, not malloc, to
>> ensure
>> that they can be unpinned safely when freed.
>>
>> This implementation will work OK for page-scale allocations, and
>> finer-grained
>> allocations will be implemented in a future patch.
>
> LGTM.
>
> Thanks,
>
> Tobias
Thank you, this one is now pushed.
Andrew
^ permalink raw reply [flat|nested] 14+ messages in thread
* Fix 'libgomp/config/linux/allocator.c' 'size_t' vs. '%ld' format string mismatch (was: Build breakage)
[not found] ` <87cyv999o7.fsf@euler.schwinge.homeip.net>
@ 2023-12-13 20:17 ` Thomas Schwinge
0 siblings, 0 replies; 14+ messages in thread
From: Thomas Schwinge @ 2023-12-13 20:17 UTC (permalink / raw)
To: Jerry D, Andrew Stubbs, gcc-patches
Cc: Tobias Burnus, Jakub Jelinek, fortran, gcc
[-- Attachment #1: Type: text/plain, Size: 2433 bytes --]
Hi!
On 2023-12-13T20:36:40+0100, I wrote:
> On 2023-12-13T11:15:54-0800, Jerry D via Gcc <gcc@gcc.gnu.org> wrote:
>> I am getting this failure to build from clean trunk.
>
> This is due to commit r14-6499-g348874f0baac0f22c98ab11abbfa65fd172f6bdd
> "libgomp: basic pinned memory on Linux", which supposedly was only tested
> with '--disable-multilib' or so. As Andrew's now on vacations --
> conveniently ;-P -- I'll soon push a fix.
Pushed to master branch commit 5445ff4a51fcee4d281f79b5f54b349290d0327d
"Fix 'libgomp/config/linux/allocator.c' 'size_t' vs. '%ld' format string mismatch",
see attached.
Grüße
Thomas
>> In file included from ../../../../trunk/libgomp/config/linux/allocator.c:31:
>> ../../../../trunk/libgomp/config/linux/allocator.c: In function
>> ‘linux_memspace_alloc’:
>> ../../../../trunk/libgomp/config/linux/allocator.c:70:26: error: format
>> ‘%ld’ expects argument of type ‘long int’, but argument 3 has type
>> ‘size_t’ {aka ‘unsigned int’} [-Werror=format=]
>> 70 | gomp_debug (0, "libgomp: failed to pin %ld bytes of"
>> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 71 | " memory (ulimit too low?)\n", size);
>> | ~~~~
>> | |
>> | size_t
>> {aka unsigned int}
>> ../../../../trunk/libgomp/libgomp.h:186:29: note: in definition of macro
>> ‘gomp_debug’
>> 186 | (gomp_debug) ((KIND), __VA_ARGS__); \
>> | ^~~~~~~~~~~
>> ../../../../trunk/libgomp/config/linux/allocator.c:70:52: note: format
>> string is defined here
>> 70 | gomp_debug (0, "libgomp: failed to pin %ld bytes of"
>> | ~~^
>> | |
>> | long int
>> | %d
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955
[-- Attachment #2: 0001-Fix-libgomp-config-linux-allocator.c-size_t-vs.-ld-f.patch --]
[-- Type: text/x-diff, Size: 3578 bytes --]
From 5445ff4a51fcee4d281f79b5f54b349290d0327d Mon Sep 17 00:00:00 2001
From: Thomas Schwinge <thomas@codesourcery.com>
Date: Wed, 13 Dec 2023 17:48:11 +0100
Subject: [PATCH] Fix 'libgomp/config/linux/allocator.c' 'size_t' vs. '%ld'
format string mismatch
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Fix-up for commit 348874f0baac0f22c98ab11abbfa65fd172f6bdd
"libgomp: basic pinned memory on Linux", which may result in build failures
as follow, for example, for the '-m32' multilib of x86_64-pc-linux-gnu:
In file included from [...]/source-gcc/libgomp/config/linux/allocator.c:31:
[...]/source-gcc/libgomp/config/linux/allocator.c: In function ‘linux_memspace_alloc’:
[...]/source-gcc/libgomp/config/linux/allocator.c:70:26: error: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘size_t’ {aka ‘unsigned int’} [-Werror=format=]
70 | gomp_debug (0, "libgomp: failed to pin %ld bytes of"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
71 | " memory (ulimit too low?)\n", size);
| ~~~~
| |
| size_t {aka unsigned int}
[...]/source-gcc/libgomp/libgomp.h:186:29: note: in definition of macro ‘gomp_debug’
186 | (gomp_debug) ((KIND), __VA_ARGS__); \
| ^~~~~~~~~~~
[...]/source-gcc/libgomp/config/linux/allocator.c:70:52: note: format string is defined here
70 | gomp_debug (0, "libgomp: failed to pin %ld bytes of"
| ~~^
| |
| long int
| %d
cc1: all warnings being treated as errors
make[9]: *** [allocator.lo] Error 1
make[9]: Leaving directory `[...]/build-gcc/x86_64-pc-linux-gnu/32/libgomp'
[...]
Fix this in the same way as used elsewhere in libgomp.
libgomp/
* config/linux/allocator.c (linux_memspace_alloc): Fix 'size_t'
vs. '%ld' format string mismatch.
---
libgomp/config/linux/allocator.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c
index 269d0d607d8..6ffa2417913 100644
--- a/libgomp/config/linux/allocator.c
+++ b/libgomp/config/linux/allocator.c
@@ -50,6 +50,9 @@
#include <sys/mman.h>
#include <string.h>
#include "libgomp.h"
+#ifdef HAVE_INTTYPES_H
+# include <inttypes.h> /* For PRIu64. */
+#endif
static void *
linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin)
@@ -67,8 +70,13 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin)
if (mlock (addr, size))
{
- gomp_debug (0, "libgomp: failed to pin %ld bytes of"
- " memory (ulimit too low?)\n", size);
+#ifdef HAVE_INTTYPES_H
+ gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes of"
+ " memory (ulimit too low?)\n", (uint64_t) size);
+#else
+ gomp_debug (0, "libgomp: failed to pin %lu bytes of"
+ " memory (ulimit too low?)\n", (unsigned long) size);
+#endif
munmap (addr, size);
return NULL;
}
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-12-13 20:17 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-12-11 17:03 [PATCH v3 0/6] libgomp: OpenMP pinned memory omp_alloc Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 1/6] libgomp: basic pinned memory on Linux Andrew Stubbs
2023-12-12 9:02 ` Tobias Burnus
2023-12-13 14:28 ` Andrew Stubbs
[not found] ` <9ffed162-2197-4acd-b72c-1946dc899f86@gmail.com>
[not found] ` <87cyv999o7.fsf@euler.schwinge.homeip.net>
2023-12-13 20:17 ` Fix 'libgomp/config/linux/allocator.c' 'size_t' vs. '%ld' format string mismatch (was: Build breakage) Thomas Schwinge
2023-12-11 17:04 ` [PATCH v3 2/6] libgomp, openmp: Add ompx_pinned_mem_alloc Andrew Stubbs
2023-12-12 10:05 ` Tobias Burnus
2023-12-12 16:17 ` Andrew Stubbs
2023-12-13 8:52 ` Jakub Jelinek
2023-12-11 17:04 ` [PATCH v3 3/6] openmp: Add -foffload-memory Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 4/6] openmp: -foffload-memory=pinned Andrew Stubbs
2023-12-11 17:04 ` [PATCH v3 5/6] libgomp, nvptx: Cuda pinned memory Andrew Stubbs
2023-12-12 9:10 ` Tobias Burnus
2023-12-11 17:04 ` [PATCH v3 6/6] libgomp: fine-grained pinned memory allocator Andrew Stubbs
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).