From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) by sourceware.org (Postfix) with ESMTPS id 93BF6385E838 for ; Fri, 31 May 2024 12:08:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 93BF6385E838 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 93BF6385E838 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::72f ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1717157293; cv=none; b=ORlMDZRTnY8D4Q8flG9Rhv5EKoIHdHw9FPqF2eZg+ViqKrH+QmDHwK/sW+4lo4dNzTVqNwf79KmI8O0OS3GvO1voM1VF0VJ9fXvyEGard1hFLvOBxCid/gjMmKeyAmmPgMB2VDb3lBv6EEVx1H/FXQBu6VydAu4hbaeuK1og+98= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1717157293; c=relaxed/simple; bh=1ZRoepvZdbepaWEP4AdflP1A+Obk2DeQ0K/I87JGgvc=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=oo3d9R76jdJEUANw6Z66g4d2UmRet6UY3B3LdNDLsUqQlHLGHxm8kp5f0iBJFiqtOLwLfTZp1GIq+C2oGhdzVcL1pud6Z4QlMbvzfjEcT3AZZ699u702LC8nuNHyTgoIiVeAbK/3oYIekGCK1jtI50aCrCNoRy/fcyQTrgt8MAk= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-qk1-x72f.google.com with SMTP id af79cd13be357-792ecce9522so125765485a.3 for ; Fri, 31 May 2024 05:08:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1717157287; x=1717762087; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=hbwyTbTqDKfu4/r4afCgQcwDBvqOr7X/8hPSLRYs73Q=; b=wh01kvGkVoNkqIB+8a6Xt4fiwCywUh8OvKvsq9el3dAmrZLr3bWfgjCmTUz2DQjHqU ZmTyKRM43pb+LkHq95FR3H+YWXRg+gNTiu0DskLTVXNaVV5PwG6O0KmTZtjJAmu7Q9cb ObiOgdKQfJCY4P7wNTduso4nolOkibnR/5LYPbLWAXrCflcoVjSL8OoorQi9ut30p0GN no1R34Mn7ico6jBSKFfu6Arfrzd+tDUeV9nJQ83V3tg+xpqmeei7WZFOqcN+D5qbHOfV uOx5f0mUXyMCTl/x5z1idfhN814SahKNMjaDbzGJxB38H7XFmf3mtNQHBemxb4fe+5gN 3dNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717157287; x=1717762087; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hbwyTbTqDKfu4/r4afCgQcwDBvqOr7X/8hPSLRYs73Q=; b=SEBAGAFK0CMEcLd3aRxstX55aVvTYZXDg05YnOmeNiFXZBDo1tai2/5NiVtY2pggEb mDoSgqI4dTw8EK7KaIX+uorpoN+IhrEpvhSw3KXdYh84byQqwj0w4AstT6lw4yIKrXFp LVSpwlyC6Sg2cEbOy9BqyiJnZ+H1BYqrtreDIInvTNH3apbpfE6s3tWZSXRhVtXEfhyA 5W2I/PKvHqWb/UY3v5NoV7F6xWNcG7PuqunwOlmoIwW+wQrRKWJEjrNJhnitwMiMi1CI lZxGyE4MosQH930VsNIuHbxEgImOaSW6VUm3SFnK5SfOQG4uSl+hUg0GRd626pfCxVSP UP0A== X-Gm-Message-State: AOJu0YxczJs8NW0nHMt9a7e0jrHKTJYNuzq4KDWliBaXijnIQJOwc0W8 7q3lkKLwS+jbyRzbSXHhyo5HVaMNKRNt10jjATHAWCoSQuK4+CpxCoUUOBzKnCAIRbLG6cbTLgQ eZvM= X-Google-Smtp-Source: AGHT+IERonLOffu+6yKdYj4TClinamLfLzctaVvyYXQWTmG5zbu4mbRSkVwfNy+BTJQYW/e3BqR/Mw== X-Received: by 2002:a05:620a:19aa:b0:790:a937:cb10 with SMTP id af79cd13be357-794f5c61814mr207850185a.7.1717157286787; Fri, 31 May 2024 05:08:06 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id d75a77b69052e-43ffa46ff45sm1134821cf.36.2024.05.31.05.08.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 May 2024 05:08:05 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Subject: [PATCH v4 4/5] libgomp, nvptx: Cuda pinned memory Date: Fri, 31 May 2024 12:07:50 +0000 Message-ID: <20240531120751.56140-5-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240531120751.56140-1-ams@baylibre.com> References: <20240531120751.56140-1-ams@baylibre.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00,DKIM_SIGNED,DKIM_VALID,GIT_PATCH_0,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,TXREP,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org List-Id: From: Thomas Schwinge This patch was already approved, by Tobias Burnus (with one caveat about initialization location), but wasn't committed at that time as I didn't want to disentangle it from the textual dependencies on the other patches in the series. -------------------- Use Cuda to pin memory, instead of Linux mlock, when available. There are two advantages: firstly, this gives a significant speed boost for NVPTX offloading, and secondly, it side-steps the usual OS ulimit/rlimit setting. The design adds a device independent plugin API for allocating pinned memory, and then implements it for NVPTX. At present, the other supported devices do not have equivalent capabilities (or requirements). libgomp/ChangeLog: * config/linux/allocator.c: Include assert.h. (using_device_for_page_locked): New variable. (linux_memspace_alloc): Add init0 parameter. Support device pinning. (linux_memspace_calloc): Set init0 to true. (linux_memspace_free): Support device pinning. (linux_memspace_realloc): Support device pinning. (MEMSPACE_ALLOC): Set init0 to false. * libgomp-plugin.h (GOMP_OFFLOAD_page_locked_host_alloc): New prototype. (GOMP_OFFLOAD_page_locked_host_free): Likewise. * libgomp.h (gomp_page_locked_host_alloc): Likewise. (gomp_page_locked_host_free): Likewise. (struct gomp_device_descr): Add page_locked_host_alloc_func and page_locked_host_free_func. * libgomp.texi: Adjust the docs for the pinned trait. * libgomp_g.h (GOMP_enable_pinned_mode): New prototype. * plugin/plugin-nvptx.c (GOMP_OFFLOAD_page_locked_host_alloc): New function. (GOMP_OFFLOAD_page_locked_host_free): Likewise. * target.c (device_for_page_locked): New variable. (get_device_for_page_locked): New function. (gomp_page_locked_host_alloc): Likewise. (gomp_page_locked_host_free): Likewise. (gomp_load_plugin_for_device): Add page_locked_host_alloc and page_locked_host_free. * testsuite/libgomp.c/alloc-pinned-1.c: Change expectations for NVPTX devices. * testsuite/libgomp.c/alloc-pinned-2.c: Likewise. * testsuite/libgomp.c/alloc-pinned-3.c: Likewise. * testsuite/libgomp.c/alloc-pinned-4.c: Likewise. * testsuite/libgomp.c/alloc-pinned-5.c: Likewise. * testsuite/libgomp.c/alloc-pinned-6.c: Likewise. Co-Authored-By: Thomas Schwinge --- libgomp/config/linux/allocator.c | 141 ++++++++++++++----- libgomp/libgomp-plugin.h | 2 + libgomp/libgomp.h | 4 + libgomp/libgomp.texi | 11 +- libgomp/libgomp_g.h | 1 + libgomp/plugin/plugin-nvptx.c | 42 ++++++ libgomp/target.c | 136 ++++++++++++++++++ libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 26 ++++ libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 26 ++++ libgomp/testsuite/libgomp.c/alloc-pinned-3.c | 45 +++++- libgomp/testsuite/libgomp.c/alloc-pinned-4.c | 44 +++++- libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 26 ++++ libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 35 ++++- 13 files changed, 489 insertions(+), 50 deletions(-) diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c index 7e09ba44b2f..063c46f972c 100644 --- a/libgomp/config/linux/allocator.c +++ b/libgomp/config/linux/allocator.c @@ -36,6 +36,11 @@ /* Implement malloc routines that can handle pinned memory on Linux. + Given that pinned memory is typically used to help host <-> device memory + transfers, we attempt to allocate such memory using a device (really: + libgomp plugin), but fall back to mmap plus mlock if no suitable device is + available. + It's possible to use mlock on any heap memory, but using munlock is problematic if there are multiple pinned allocations on the same page. Tracking all that manually would be possible, but adds overhead. This may @@ -49,6 +54,7 @@ #define _GNU_SOURCE #include #include +#include #include "libgomp.h" #ifdef HAVE_INTTYPES_H # include /* For PRIu64. */ @@ -68,50 +74,92 @@ GOMP_enable_pinned_mode () always_pinned_mode = true; } +static int using_device_for_page_locked + = /* uninitialized */ -1; + static void * -linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin) +linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, + bool init0) { - (void)memspace; + gomp_debug (0, "%s: memspace=%llu, size=%llu, pin=%d, init0=%d\n", + __FUNCTION__, (unsigned long long) memspace, + (unsigned long long) size, pin, init0); + + void *addr; /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (pin) { - /* Note that mmap always returns zeroed memory and is therefore also a - suitable implementation of calloc. */ - void *addr = mmap (NULL, size, PROT_READ | PROT_WRITE, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - if (addr == MAP_FAILED) - return NULL; - - if (mlock (addr, size)) + int using_device + = __atomic_load_n (&using_device_for_page_locked, + MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d\n", + using_device); + if (using_device != 0) + { + using_device = gomp_page_locked_host_alloc (&addr, size); + int using_device_old + = __atomic_exchange_n (&using_device_for_page_locked, + using_device, MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d, using_device_old=%d\n", + using_device, using_device_old); + assert (using_device_old == -1 + /* We shouldn't have concurrently changed our mind. */ + || using_device_old == using_device); + } + if (using_device == 0) { + gomp_debug (0, " mmap\n"); + addr = mmap (NULL, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (addr == MAP_FAILED) + addr = NULL; + else + { + /* 'mmap' zero-initializes. */ + init0 = false; + + gomp_debug (0, " mlock\n"); + if (mlock (addr, size)) + { #ifdef HAVE_INTTYPES_H - gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes of" - " memory (ulimit too low?)\n", (uint64_t) size); + gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes of" + " memory (ulimit too low?)\n", (uint64_t) size); #else - gomp_debug (0, "libgomp: failed to pin %lu bytes of" - " memory (ulimit too low?)\n", (unsigned long) size); + gomp_debug (0, "libgomp: failed to pin %lu bytes of memory" + " (ulimit too low?)\n", (unsigned long) size); #endif - munmap (addr, size); - return NULL; + munmap (addr, size); + addr = NULL; + } + } } - - return addr; } else - return malloc (size); + addr = malloc (size); + + if (addr && init0) + { + gomp_debug (0, " init0\n"); + memset (addr, 0, size); + } + + return addr; } static void * linux_memspace_calloc (omp_memspace_handle_t memspace, size_t size, int pin) { + gomp_debug (0, "%s: memspace=%llu, size=%llu, pin=%d\n", + __FUNCTION__, (unsigned long long) memspace, (unsigned long long) size, pin); + /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (pin) - return linux_memspace_alloc (memspace, size, pin); + return linux_memspace_alloc (memspace, size, pin, true); else return calloc (1, size); } @@ -120,13 +168,25 @@ static void linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size, int pin) { - (void)memspace; + gomp_debug (0, "%s: memspace=%llu, addr=%p, size=%llu, pin=%d\n", + __FUNCTION__, (unsigned long long) memspace, addr, (unsigned long long) size, pin); /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (pin) - munmap (addr, size); + { + int using_device + = __atomic_load_n (&using_device_for_page_locked, + MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d\n", + using_device); + if (using_device == 1) + gomp_page_locked_host_free (addr); + else + /* 'munlock'ing is implicit with following 'munmap'. */ + munmap (addr, size); + } else free (addr); } @@ -135,11 +195,25 @@ static void * linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, size_t oldsize, size_t size, int oldpin, int pin) { + gomp_debug (0, "%s: memspace=%llu, addr=%p, oldsize=%llu, size=%llu, oldpin=%d, pin=%d\n", + __FUNCTION__, (unsigned long long) memspace, addr, (unsigned long long) oldsize, (unsigned long long) size, oldpin, pin); + /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (oldpin && pin) { + /* We can only expect to be able to just 'mremap' if not using a device + for page-locked memory. */ + int using_device + = __atomic_load_n (&using_device_for_page_locked, + MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d\n", + using_device); + if (using_device != 0) + goto manual_realloc; + + gomp_debug (0, " mremap\n"); void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE); if (newaddr == MAP_FAILED) return NULL; @@ -147,18 +221,19 @@ linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, return newaddr; } else if (oldpin || pin) - { - void *newaddr = linux_memspace_alloc (memspace, size, pin); - if (newaddr) - { - memcpy (newaddr, addr, oldsize < size ? oldsize : size); - linux_memspace_free (memspace, addr, oldsize, oldpin); - } - - return newaddr; - } + goto manual_realloc; else return realloc (addr, size); + +manual_realloc:; + void *newaddr = linux_memspace_alloc (memspace, size, pin, false); + if (newaddr) + { + memcpy (newaddr, addr, oldsize < size ? oldsize : size); + linux_memspace_free (memspace, addr, oldsize, oldpin); + } + + return newaddr; } static int @@ -169,7 +244,7 @@ linux_memspace_validate (omp_memspace_handle_t, unsigned, int) } #define MEMSPACE_ALLOC(MEMSPACE, SIZE, PIN) \ - linux_memspace_alloc (MEMSPACE, SIZE, PIN) + linux_memspace_alloc (MEMSPACE, SIZE, PIN, false) #define MEMSPACE_CALLOC(MEMSPACE, SIZE, PIN) \ linux_memspace_calloc (MEMSPACE, SIZE, PIN) #define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN) \ diff --git a/libgomp/libgomp-plugin.h b/libgomp/libgomp-plugin.h index 0c9c28c65cf..0d2e3f0a6ec 100644 --- a/libgomp/libgomp-plugin.h +++ b/libgomp/libgomp-plugin.h @@ -139,6 +139,8 @@ extern int GOMP_OFFLOAD_load_image (int, unsigned, const void *, extern bool GOMP_OFFLOAD_unload_image (int, unsigned, const void *); extern void *GOMP_OFFLOAD_alloc (int, size_t); extern bool GOMP_OFFLOAD_free (int, void *); +extern bool GOMP_OFFLOAD_page_locked_host_alloc (void **, size_t); +extern bool GOMP_OFFLOAD_page_locked_host_free (void *); extern bool GOMP_OFFLOAD_dev2host (int, void *, const void *, size_t); extern bool GOMP_OFFLOAD_host2dev (int, void *, const void *, size_t); extern bool GOMP_OFFLOAD_dev2dev (int, void *, const void *, size_t); diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h index 089393846d1..b8c6fc00de4 100644 --- a/libgomp/libgomp.h +++ b/libgomp/libgomp.h @@ -1128,6 +1128,8 @@ extern int gomp_get_num_devices (void); extern bool gomp_target_task_fn (void *); extern void gomp_target_rev (uint64_t, uint64_t, uint64_t, uint64_t, uint64_t, int, struct goacc_asyncqueue *); +extern bool gomp_page_locked_host_alloc (void **, size_t); +extern void gomp_page_locked_host_free (void *); /* Splay tree definitions. */ typedef struct splay_tree_node_s *splay_tree_node; @@ -1409,6 +1411,8 @@ struct gomp_device_descr __typeof (GOMP_OFFLOAD_unload_image) *unload_image_func; __typeof (GOMP_OFFLOAD_alloc) *alloc_func; __typeof (GOMP_OFFLOAD_free) *free_func; + __typeof (GOMP_OFFLOAD_page_locked_host_alloc) *page_locked_host_alloc_func; + __typeof (GOMP_OFFLOAD_page_locked_host_free) *page_locked_host_free_func; __typeof (GOMP_OFFLOAD_dev2host) *dev2host_func; __typeof (GOMP_OFFLOAD_host2dev) *host2dev_func; __typeof (GOMP_OFFLOAD_memcpy2d) *memcpy2d_func; diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi index aa0a81062ec..bdd0b1550db 100644 --- a/libgomp/libgomp.texi +++ b/libgomp/libgomp.texi @@ -6285,8 +6285,9 @@ a @code{nearest} allocation. Additional notes regarding the traits: @itemize -@item The @code{pinned} trait is supported on Linux hosts, but is subject to - the OS @code{ulimit}/@code{rlimit} locked memory settings. +@item The @code{pinned} trait is supported on Linux hosts, but is usually + subject to the OS @code{ulimit}/@code{rlimit} locked memory settings (see + @ref{Offload-Target Specifics} for exceptions). @item The default for the @code{pool_size} trait is no pool and for every (re)allocation the associated library routine is called, which might internally use a memory pool. @@ -6474,6 +6475,12 @@ The implementation remark: @code{omp_thread_mem_alloc}, all use low-latency memory as first preference, and fall back to main graphics memory when the low-latency pool is exhausted. +@item Pinned memory allocated using @code{omp_alloc} with the + @code{ompx_pinned_mem_alloc} allocator or the @code{pinned} trait is + obtained via the CUDA API when an NVPTX device is present. This provides + a performance boost for NVPTX offload code and also allows unlimited use + of pinned memory regardless of the OS @code{ulimit}/@code{rlimit} + settings. @end itemize diff --git a/libgomp/libgomp_g.h b/libgomp/libgomp_g.h index c0cc03ae61f..74c0fd17784 100644 --- a/libgomp/libgomp_g.h +++ b/libgomp/libgomp_g.h @@ -371,6 +371,7 @@ extern bool GOMP_is_alloc (void *); extern void *GOMP_alloc (size_t, size_t, uintptr_t); extern void GOMP_free (void *, uintptr_t); +extern void GOMP_enable_pinned_mode (void); /* error.c */ diff --git a/libgomp/plugin/plugin-nvptx.c b/libgomp/plugin/plugin-nvptx.c index 4cedc5390a3..5caf8fe5026 100644 --- a/libgomp/plugin/plugin-nvptx.c +++ b/libgomp/plugin/plugin-nvptx.c @@ -1625,6 +1625,48 @@ GOMP_OFFLOAD_free (int ord, void *ptr) && nvptx_free (ptr, ptx_devices[ord])); } +bool +GOMP_OFFLOAD_page_locked_host_alloc (void **ptr, size_t size) +{ + GOMP_PLUGIN_debug (0, "nvptx %s: ptr=%p, size=%llu\n", + __FUNCTION__, ptr, (unsigned long long) size); + + if (size == 0) + { + /* Special case to ensure omp_alloc specification compliance. */ + *ptr = NULL; + GOMP_PLUGIN_debug (0, " -> *ptr=null\n"); + return true; + } + + CUresult r; + + unsigned int flags = 0; + /* Given 'CU_DEVICE_ATTRIBUTE_UNIFIED_ADDRESSING', we don't need + 'flags |= CU_MEMHOSTALLOC_PORTABLE;' here. */ + r = CUDA_CALL_NOCHECK (cuMemHostAlloc, ptr, size, flags); + if (r == CUDA_ERROR_OUT_OF_MEMORY) + *ptr = NULL; + else if (r != CUDA_SUCCESS) + { + GOMP_PLUGIN_error ("cuMemHostAlloc error: %s", cuda_error (r)); + return false; + } + GOMP_PLUGIN_debug (0, " -> *ptr=%p\n", + *ptr); + return true; +} + +bool +GOMP_OFFLOAD_page_locked_host_free (void *ptr) +{ + GOMP_PLUGIN_debug (0, "nvptx %s: ptr=%p\n", + __FUNCTION__, ptr); + + CUDA_CALL (cuMemFreeHost, ptr); + return true; +} + void GOMP_OFFLOAD_openacc_exec (void (*fn) (void *), size_t mapnum __attribute__((unused)), diff --git a/libgomp/target.c b/libgomp/target.c index 48689920d4a..adbd18101f2 100644 --- a/libgomp/target.c +++ b/libgomp/target.c @@ -4437,6 +4437,140 @@ omp_target_free (void *device_ptr, int device_num) gomp_mutex_unlock (&devicep->lock); } +/* Device (really: libgomp plugin) to use for paged-locked memory. We + assume there is either none or exactly one such device for the lifetime of + the process. */ + +static struct gomp_device_descr *device_for_page_locked + = /* uninitialized */ (void *) -1; + +static struct gomp_device_descr * +get_device_for_page_locked (void) +{ + gomp_debug (0, "%s\n", + __FUNCTION__); + + struct gomp_device_descr *device; +#ifdef HAVE_SYNC_BUILTINS + device + = __atomic_load_n (&device_for_page_locked, MEMMODEL_RELAXED); + if (device == (void *) -1) + { + gomp_debug (0, " init\n"); + + gomp_init_targets_once (); + + device = NULL; + for (int i = 0; i < num_devices; ++i) + { + gomp_debug (0, " i=%d, target_id=%d\n", + i, devices[i].target_id); + + /* We consider only the first device of potentially several of the + same type as this functionality is not specific to an individual + offloading device, but instead relates to the host-side + implementation of the respective offloading implementation. */ + if (devices[i].target_id != 0) + continue; + + if (!devices[i].page_locked_host_alloc_func) + continue; + + gomp_debug (0, " found device: %p (%s)\n", + &devices[i], devices[i].name); + if (device) + gomp_fatal ("Unclear how %s and %s libgomp plugins may" + " simultaneously provide functionality" + " for page-locked memory", + device->name, devices[i].name); + else + device = &devices[i]; + } + + struct gomp_device_descr *device_old + = __atomic_exchange_n (&device_for_page_locked, device, + MEMMODEL_RELAXED); + gomp_debug (0, " old device_for_page_locked: %p\n", + device_old); + assert (device_old == (void *) -1 + /* We shouldn't have concurrently found a different or no + device. */ + || device_old == device); + } +#else /* !HAVE_SYNC_BUILTINS */ + gomp_debug (0, " not implemented for '!HAVE_SYNC_BUILTINS'\n"); + (void) &device_for_page_locked; + device = NULL; +#endif /* HAVE_SYNC_BUILTINS */ + + gomp_debug (0, " -> device=%p (%s)\n", + device, device ? device->name : "[none]"); + return device; +} + +/* Allocate page-locked host memory. + Returns whether we have a device capable of that. */ + +attribute_hidden bool +gomp_page_locked_host_alloc (void **ptr, size_t size) +{ + gomp_debug (0, "%s: ptr=%p, size=%llu\n", + __FUNCTION__, ptr, (unsigned long long) size); + + struct gomp_device_descr *device = get_device_for_page_locked (); + gomp_debug (0, " device=%p (%s)\n", + device, device ? device->name : "[none]"); + if (device) + { + gomp_mutex_lock (&device->lock); + if (device->state == GOMP_DEVICE_UNINITIALIZED) + gomp_init_device (device); + else if (device->state == GOMP_DEVICE_FINALIZED) + { + gomp_mutex_unlock (&device->lock); + gomp_fatal ("Device %s used for for page-locked memory is finalized", + device->name); + } + gomp_mutex_unlock (&device->lock); + + if (!device->page_locked_host_alloc_func (ptr, size)) + gomp_fatal ("Failed to allocate page-locked host memory" + " via %s libgomp plugin", + device->name); + } + return device != NULL; +} + +/* Free page-locked host memory. + This must only be called if 'gomp_page_locked_host_alloc' returned + 'true'. */ + +attribute_hidden void +gomp_page_locked_host_free (void *ptr) +{ + gomp_debug (0, "%s: ptr=%p\n", + __FUNCTION__, ptr); + + struct gomp_device_descr *device = get_device_for_page_locked (); + gomp_debug (0, " device=%p (%s)\n", + device, device ? device->name : "[none]"); + assert (device); + + gomp_mutex_lock (&device->lock); + assert (device->state != GOMP_DEVICE_UNINITIALIZED); + if (device->state == GOMP_DEVICE_FINALIZED) + { + gomp_mutex_unlock (&device->lock); + return; + } + gomp_mutex_unlock (&device->lock); + + if (!device->page_locked_host_free_func (ptr)) + gomp_fatal ("Failed to free page-locked host memory" + " via %s libgomp plugin", + device->name); +} + int omp_target_is_present (const void *ptr, int device_num) { @@ -5159,6 +5293,8 @@ gomp_load_plugin_for_device (struct gomp_device_descr *device, DLSYM (unload_image); DLSYM (alloc); DLSYM (free); + DLSYM_OPT (page_locked_host_alloc, page_locked_host_alloc); + DLSYM_OPT (page_locked_host_free, page_locked_host_free); DLSYM (dev2host); DLSYM (host2dev); DLSYM_OPT (memcpy2d, memcpy2d); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c index 4185accf2e6..71cad052a4d 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c @@ -2,6 +2,8 @@ /* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory works. */ #include @@ -71,10 +73,16 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, allowing space for overhead, but stay within the ulimit. */ const int SIZE = PAGE_SIZE - 128; CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures +#endif const omp_alloctrait_t traits[] = { { omp_atk_pinned, 1 } @@ -96,21 +104,39 @@ main () abort (); int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else if (amount == 0) abort (); +#endif p = omp_realloc (p, SIZE * 2, allocator, allocator); int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else if (amount2 <= amount) abort (); +#endif /* SIZE*2 ensures that it doesn't slot into the space possibly vacated by realloc. */ p = omp_calloc (1, SIZE * 2, allocator); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else if (get_pinned_mem () <= amount2) abort (); +#endif verify0 (p, SIZE * 2); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c index 0b9c11d0315..84356785eca 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c @@ -2,6 +2,8 @@ /* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory works (pool_size code path). */ #include @@ -71,10 +73,16 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, allowing space for overhead, but stay within the ulimit. */ const int SIZE = PAGE_SIZE - 128; CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures +#endif const omp_alloctrait_t traits[] = { { omp_atk_pinned, 1 }, @@ -97,16 +105,28 @@ main () abort (); int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else if (amount == 0) abort (); +#endif p = omp_realloc (p, SIZE * 2, allocator, allocator); if (!p) abort (); int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else if (amount2 <= amount) abort (); +#endif /* SIZE*2 ensures that it doesn't slot into the space possibly vacated by realloc. */ @@ -114,8 +134,14 @@ main () if (!p) abort (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else if (get_pinned_mem () <= amount2) abort (); +#endif verify0 (p, SIZE * 2); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-3.c b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c index 53e4720cc9c..b17daf7e222 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-3.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c @@ -1,5 +1,7 @@ /* { dg-do run } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory fails correctly. */ #include @@ -75,8 +77,15 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* This needs to be large enough to cover multiple pages. */ const int SIZE = PAGE_SIZE * 4; +#endif + const int PIN_LIMIT = PAGE_SIZE * 2; /* Pinned memory, no fallback. */ const omp_alloctrait_t traits1[] = { @@ -101,23 +110,34 @@ main () #endif /* Ensure that the limit is smaller than the allocation. */ - set_pin_limit (SIZE / 2); + set_pin_limit (PIN_LIMIT); // Sanity check if (get_pinned_mem () != 0) abort (); - // Should fail void *p1 = omp_alloc (SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p1) + abort (); +#else + // Should fail if (p1) abort (); +#endif - // Should fail void *p2 = omp_calloc (1, SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p2) + abort (); +#else + // Should fail if (p2) abort (); +#endif - // Should fall back void *p3 = omp_alloc (SIZE, allocator2); if (!p3) abort (); @@ -128,16 +148,29 @@ main () abort (); verify0 (p4, SIZE); - // Should fail to realloc void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc); void *p5 = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'; does reallocate. + if (!notpinned || !p5 || p5 == notpinned) + abort (); +#else + // Should fail to realloc if (!notpinned || p5) abort (); +#endif - // Should fall back to no realloc needed +#ifdef OFFLOAD_DEVICE_NVPTX + void *p6 = omp_realloc (p5, SIZE, allocator2, allocator1); + // Does reallocate. + if (p5 == p6) + abort (); +#else void *p6 = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc); + // Should fall back to no realloc needed if (p6 != notpinned) abort (); +#endif // No memory should have been pinned int amount = get_pinned_mem (); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-4.c b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c index 9d850c23e4b..aadd9249ba9 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-4.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c @@ -1,5 +1,7 @@ /* { dg-do run } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory fails correctly, pool_size code path. */ #include @@ -75,8 +77,15 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* This needs to be large enough to cover multiple pages. */ const int SIZE = PAGE_SIZE * 4; +#endif + const int PIN_LIMIT = PAGE_SIZE * 2; /* Pinned memory, no fallback. */ const omp_alloctrait_t traits1[] = { @@ -103,21 +112,33 @@ main () #endif /* Ensure that the limit is smaller than the allocation. */ - set_pin_limit (SIZE / 2); + set_pin_limit (PIN_LIMIT); // Sanity check if (get_pinned_mem () != 0) abort (); - // Should fail void *p = omp_alloc (SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif - // Should fail p = omp_calloc (1, SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif // Should fall back p = omp_alloc (SIZE, allocator2); @@ -130,16 +151,29 @@ main () abort (); verify0 (p, SIZE); - // Should fail to realloc void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc); p = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'; does reallocate. + if (!notpinned || !p || p == notpinned) + abort (); +#else + // Should fail to realloc if (!notpinned || p) abort (); +#endif - // Should fall back to no realloc needed +#ifdef OFFLOAD_DEVICE_NVPTX + void *p_ = omp_realloc (p, SIZE, allocator2, allocator1); + // Does reallocate. + if (p_ == p) + abort (); +#else p = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc); + // Should fall back to no realloc needed if (p != notpinned) abort (); +#endif // No memory should have been pinned int amount = get_pinned_mem (); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c index 18e6d20ca5b..95e4562537c 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c @@ -2,6 +2,8 @@ /* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that ompx_pinned_mem_alloc works. */ #include @@ -67,10 +69,16 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, allowing space for overhead, but stay within the ulimit. */ const int SIZE = PAGE_SIZE - 128; CHECK_SIZE (SIZE * 5); +#endif // Sanity check if (get_pinned_mem () != 0) @@ -81,21 +89,39 @@ main () abort (); int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else if (amount == 0) abort (); +#endif p = omp_realloc (p, SIZE * 2, ompx_pinned_mem_alloc, ompx_pinned_mem_alloc); int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else if (amount2 <= amount) abort (); +#endif /* SIZE*2 ensures that it doesn't slot into the space possibly vacated by realloc. */ p = omp_calloc (1, SIZE * 2, ompx_pinned_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else if (get_pinned_mem () <= amount2) abort (); +#endif verify0 (p, SIZE * 2); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c index f80a0264f97..aefb69a1d60 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c @@ -1,5 +1,7 @@ /* { dg-do run } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that ompx_pinned_mem_alloc fails correctly. */ #include @@ -66,31 +68,56 @@ set_pin_limit () int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, but stay within the ulimit. */ const int SIZE = PAGE_SIZE * 4; +#endif + const int PIN_LIMIT = PAGE_SIZE*2; /* Ensure that the limit is smaller than the allocation. */ - set_pin_limit (SIZE / 2); + set_pin_limit (PIN_LIMIT); // Sanity check if (get_pinned_mem () != 0) abort (); - // Should fail void *p = omp_alloc (SIZE, ompx_pinned_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif - // Should fail p = omp_calloc (1, SIZE, ompx_pinned_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif - // Should fail to realloc void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc); p = omp_realloc (notpinned, SIZE, ompx_pinned_mem_alloc, omp_default_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'; does reallocate. + if (!notpinned || !p || p == notpinned) + abort (); +#else + // Should fail to realloc if (!notpinned || p) abort (); +#endif // No memory should have been pinned int amount = get_pinned_mem (); -- 2.41.0