Re: [PATCH] libgomp, openmp: pinned memory

[Thomas Schwinge <thomas@codesourcery.com>]

Hi!

Re OpenMP 'pinned' memory allocator trait semantics vs. 'omp_realloc':

On 2022-01-13T13:53:03+0000, Andrew Stubbs <ams@codesourcery.com> wrote:
> On 05/01/2022 17:07, Andrew Stubbs wrote:
>> [...], I'm working on an implementation using mmap instead of malloc
>> for pinned allocations.  [...]

> This means that large allocations will now be page aligned and therefore
> pin the smallest number of pages for the size requested, and that that
> memory will be unpinned automatically when freed via munmap, or moved
> via mremap.

> --- /dev/null
> +++ b/libgomp/config/linux/allocator.c

> +static void *
> +linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr,
> +                     size_t oldsize, size_t size, int oldpin, int pin)
> +{
> +  if (oldpin && pin)
> +    {
> +      void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE);
> +      if (newaddr == MAP_FAILED)
> +     return NULL;
> +
> +      return newaddr;
> +    }
> +  else if (oldpin || pin)
> +    {
> +      void *newaddr = linux_memspace_alloc (memspace, size, pin);
> +      if (newaddr)
> +     {
> +       memcpy (newaddr, addr, oldsize < size ? oldsize : size);
> +       linux_memspace_free (memspace, addr, oldsize, oldpin);
> +     }
> +
> +      return newaddr;
> +    }
> +  else
> +    return realloc (addr, size);
> +}

I did wonder if 'mremap' with 'MREMAP_MAYMOVE' is really acceptable here,
given OpenMP 5.2, 6.2 "Memory Allocators": "Allocators with the 'pinned'
trait defined to be 'true' ensure that their allocations remain in the
same storage resource at the same location for their entire lifetime."
I'd have read into this that 'realloc' may shrink or enlarge the region
(unless even that considered faulty), but the region must not be moved
("same location"), thus no 'MREMAP_MAYMOVE'; see 'man 2 mremap'
(2019-03-06):

    'MREMAP_MAYMOVE'
        By  default, if there is not sufficient space to expand a mapping at its current location, then 'mremap()' fails.  If this flag is specified, then the kernel is permitted to relocate the mapping to a new virtual address, if necessary.  If the mapping is relocated, then absolute pointers into the old mapping location become invalid (offsets relative to the starting address of the mapping should be employed).

..., but then I saw that OpenMP 5.2, 18.13.9 'omp_realloc' is specified
such that it isn't expected to 'realloc' in-place, but rather it
"deallocates previously allocated memory and requests a memory
allocation", which I understand that it does end a "lifetime" and then
establish a new "lifetime", which means that 'MREMAP_MAYMOVE' in fact is
fine (as implemented)?


Further I read in 'man 2 mremap' (2019-03-06):

    If  the  memory segment specified by *old_address* and *old_size* is locked (using 'mlock(2)' or similar), then this lock is maintained when the segment is resized and/or relocated.  As a consequence, the amount of memory locked by the process may change.

(The current proposed code evidently does make use of that; OK.)

But then in 'NOTES' I read:

    If 'mremap()' is used to move or expand an area locked with 'mlock(2)' or equivalent, the 'mremap()' call will make a best effort to populate the new area but will not fail with 'ENOMEM' if the area cannot be populated.

What exactly is that supposed to tell us: "will make a best effort [...]
but will not fail"?  Isn't that in conflict with the earlier statement?
So can we rely on 'mremap' together with 'mlock' or not?


(This topic remains valid even if we follow through the idea of using
CUDA to register page-locked memory, because that's not available in all
configurations, and we then still want to do the 'mmap'/'mlock' thing, I
suppose.)


Grüße
 Thomas
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955