Re: Update on shared memory coarrays

[Iain Sandoe <iain@sandoe.co.uk>]

[-- Attachment #1: Type: text/plain, Size: 4112 bytes --]

Hi folks,

Nicolas König <koenigni@student.ethz.ch> wrote:
> On 04/01/2021 03:34, Iain Sandoe wrote:
>> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
>>> Am 03.01.21 um 16:42 schrieb Iain Sandoe:

>> It seems shm is not a favoured IPC mechanism for Darwin (it’s possibly  
>> disallowed for GUI apps, according to search results - I didn’t check  
>> that)
>>>> (we can file a radar, and maybe it gets fixed at some point - but  
>>>> that’s no help to all the existing system versions)
>>>
>>> Makes sense to file a bug report with Apple.  And yes, we'll have to
>>> deal with this some other way.
>> [for command line apps] Either mechanism for accessing shm works;
>> [1] shm_open / fd / mmap / munmap
>> [2] shmget / shmat / shmdt <= allocation is rounded up to the nearest  
>> page boundary.
>> Both have the constraint [on Darwin] that the size of the allocation  
>> cannot be increased after it is set (by ftruncate() [1] or shmget() [2]).
>> It can be decreased (by [2] only), but it’s not clear if that actually  
>> has any physical effect.
>
> That's a problem, since coarrays can be allocated at runtime, and we  
> can't possibly precompute their size.

OK.

> To get coarrays to work, we would probably need to create new shared  
> memory objects (with predictable names, so they can be opened from other  
> images), but that would be a significant time investment.

How does it work between different images with shared memory unless the  
name to shm_open () is predictable between images?

(there has to be some shared key of some form, right?)

> It also doesn't bring my hopes up that other things like  
> pthread_{mutex,cond}attr_setpshared() will work.

Darwin is Posix SUSv3 (actually, AFAICT the majority of the certified  
platforms are; there only seems to be one certified to UNIX7).

- so pthreads stuff that’s mandated in SUSv3 is definitely present on Darwin.

(in practice some of the optional stuff is also present - you can look at  
what libstdc++ does to get an idea of what works in a cross-platform manner)

> I think it's probably better to focus the attention on getting it to work  
> on Linux/BSD.

Well, I can understand making progress in that way - but it’s a bit of a  
slippery slope towards an “x86 Linux only” compiler (if we keep taking that  
route until a significant portion of the useful features don’t work  
elsewhere).

Of course, this particular case is not so exclusive [it appears from  
previous mails that Darwin is the only one with an issue], but as a general  
rule it would be good to pick an impl that works on at least the regularly  
used/tested targets.

>> I guess that one ought to consider that a change in allocation could  
>> trigger a copy on any system (since there might not be contiguous memory  
>> pages available for an arbitrary increase) - so a strategy which either  
>> allocates separae fragments for each need - or waits until the size is  
>> known before allocation is likely to perform better on average,
>
> There's no need to have the continuous virtual memory backed by  
> continuous physical memory, so nothing here should trigger a relocation  
> of the actual physical memory. There are just some entries in the page  
> table pointing to the same pages.

I guess if everything is done on page boundaries & sizes, that can work -  
providing one doesn’t mind the base pointer changing across resizes.

====

Update:

three codes attached.

1. Posix shm implementation; works fine but is not resizeable.

2. Darwin shm implementation : works fine but is only resizable downwards  
and it’s not clear if the resources are actually released.

3. mmapped shared file : This works [on Darwin at least] and is resizable.   
I would assume that, in reality, all accesses would be to the buffer cache  
[unless something actually forces the cache to be flushed] and therefore  
that the performance ought to be similar to shm.

Worst case, one could point /tmp at a ramdisk ..
(I tend to do that anyway, I don’t want my sshds getting hammered by  
toolchain builds).

Thoughts?
Iain


[-- Attachment #2: shm-posix.c --]
[-- Type: application/octet-stream, Size: 3252 bytes --]

/* shared memory IPC using Posix shm_open/ftruncate/mmap.
   The Darwin implementation does not appear to allow a shared memory segment
   to be resized once the initial size is set.
*/

#include <stdbool.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/shm.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>

#define NAME "/tmp/random_namexxx"

typedef struct shared_res
{
  void *ptr;
  size_t size;
  int fd;
  int status;
} shared_res_t;

shared_res_t
get_shared_resource (size_t size, bool master)
{
  shared_res_t res = {0};

  int fd;
  if (master)
    /* Create the segment.  */
   fd = shm_open (NAME, O_CREAT | O_RDWR,  S_IRUSR | S_IWUSR);
  else
    /* Wait for the segment to be created.  */
    while ((fd = shm_open (NAME, O_RDWR,  S_IRUSR | S_IWUSR)) < 0 
	   && errno == ENOENT) usleep (100000);

  if (fd < 0)
    {
      perror ("get_shared_resource: open failed");
      goto fatal_err;
    }
  else
    printf ("get_shared_resource: fd = %d\n", fd);

  if (master)
    {
      if (ftruncate (fd, size) < 0)
	{
	  perror ("get_shared_resource: ftruncate failed");
  	  goto fatal_err;
	}
    }

  void *p1 = mmap (NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : mmap failed");
      goto fatal_err;
    }
  res.fd = fd;
  res.ptr = p1;
  res.size = size;
  res.status = 0;
  return res;

fatal_err:
  if (fd >= 0)
    close (fd);
  shm_unlink (NAME);
  exit (EXIT_FAILURE);  
}

void
resize_shared_resource (shared_res_t *r, size_t new_size)
{
  if (r->status != 0)
    {
      perror ("resize_shared_resource : bad shared resource status");
      goto fatal_err;
    }

  if (ftruncate (r->fd, new_size) < 0)
    {
      perror ("resize_shared_resource: ftruncate failed");
      goto fatal_err;
    }

  /* TODO: remap this if we succeeded ... but we do not*/
  return;

fatal_err:
  if (r->fd >= 0)
    close (r->fd);
  shm_unlink (NAME);
  exit (EXIT_FAILURE);  
}

void release_shared_resource (shared_res_t r)
{
  if (munmap (r.ptr, r.size) < 0 )
    {
      perror ("release_shared_resource : unmap failed");
    }
  shm_unlink (NAME);
  close (r.fd);
}

#if SENDER

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("sender page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, true);
  printf ("sender shared = 0x%p\n", shared.ptr);
  int *ip = shared.ptr;
  *ip = 42;

  while (*ip == 42)
    usleep (100000);

  /* Any attempt to re-size this (increase or decrease) fails.
  resize_shared_resource (&shared, pagesize * 2);
  */
  release_shared_resource (shared);
  return 0;
}

#else

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("client page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, false);
  printf ("client shared = 0x%p\n", shared.ptr);

  int *ip = shared.ptr;

  while (*ip != 42)
    usleep (100000);

  *ip = 6174;

  release_shared_resource (shared);
  return 0;
}

#endif

[-- Attachment #3: shm-darwin.c --]
[-- Type: application/octet-stream, Size: 3892 bytes --]

/* shared memory IPC using shmget/shmat/shmdt.
   There doesn't appear to be any mechanism to re-size such a segment
   (although re-getting with a smaller size produces no error, it doesn't
    make the segment size contained in the status any smaller).
*/

#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/shm.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>

typedef struct shared_res
{
  void *ptr;
  size_t size;
  int smid;
  int status;
  key_t savedKey;
} shared_res_t;

shared_res_t
get_shared_resource (size_t size)
{
  shared_res_t res = {0};

  key_t myKey = ftok ("/dev/null", 6174);
  if (myKey < 0)
    {
      perror ("ftok failed");
      exit (EXIT_FAILURE);
    }

  int smid = shmget (myKey, size, IPC_CREAT | SHM_R | SHM_W);
  if (smid < 0)
    {
      perror ("shmget failed");
      exit (EXIT_FAILURE);
    }
  printf ("smid = %d\n", smid);

  void *p1 = shmat (smid, NULL, SHM_R | SHM_W);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : shmat failed");
      exit (EXIT_FAILURE);
    }
  res.smid = smid;
  res.ptr = p1;
  res.size = size;
  res.status = 0;
  res.savedKey = myKey;
  return res;
}

void
resize_shared_resource (shared_res_t *r, size_t new_size)
{
  struct shmid_ds sb;
  int res = shmctl(r->smid, IPC_STAT, &sb);
  if (res < 0)
    {
      perror ("shmctl (1) failed");
      exit (EXIT_FAILURE);
    }
  printf ("size in shm = %zu\n", sb.shm_segsz);

  /* An attempt to get the same segment with a different size fails.
     detach and start again.  */
  if (shmdt (r->ptr) < 0 )
    {
      perror ("resize_shared_resource : shmdt failed");
    }

  int smid1 = shmget (r->savedKey, new_size, IPC_CREAT | SHM_R | SHM_W);
  if (smid1 < 0)
    {
      perror ("shmget (2) failed");
    }
  printf ("smid1 = %d\n", smid1);
  void *p1 = shmat (smid1, NULL, SHM_R | SHM_W);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : shmat failed");
      exit (EXIT_FAILURE);
    }
  r->smid = smid1;
  r->ptr = p1;
  r->size = new_size;  
}

void release_shared_resource (shared_res_t r)
{
  /* detach.  */
  if (shmdt (r.ptr) < 0 )
    perror ("release_shared_resource : shmdt failed");

  struct shmid_ds sb;
  int res = shmctl(r.smid, IPC_STAT, &sb);
  if (res < 0)
    {
      perror ("release_shared_resource : shmctl failed");
      return;
    }

  printf ("release_shared_resource : size in shm = %zu\n", sb.shm_segsz);
  /* If this is the last use, then remove the shm segment.  */
  if (sb.shm_nattch == 0)
    {
      res = shmctl(r.smid, IPC_RMID, &sb);
      if (res < 0)
	{
	  perror ("release_shared_resource : shmctl (IPC_RMID) failed");
	}
    } 
}

#if SENDER

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("sender page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize * 2);
  printf ("sender shared = 0x%p\n", shared.ptr);
  int *ip = shared.ptr;
  *ip = 42;

  while (*ip == 42)
    usleep (100000);

  /* A resize to a smaller segment proceeds without error, but the actual
     reported size from the shm segment is not changed.  */

  printf ("sender attempting resize to %zu\n", pagesize);
  resize_shared_resource (&shared, pagesize);
  printf ("sender releasing\n");
  release_shared_resource (shared);
  return 0;
}

#else

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("client page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize * 2);
  printf ("client shared = 0x%p\n", shared.ptr);

  int *ip = shared.ptr;

  while (*ip != 42)
    usleep (100000);

  *ip = 6174;

  printf ("client releasing\n");
  release_shared_resource (shared);
  return 0;
}

#endif

[-- Attachment #4: ram-file.c --]
[-- Type: application/octet-stream, Size: 3545 bytes --]

/* shared resource IPC using mmapped file.
   This allows resizing.
*/

#include <stdbool.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/mman.h>
#include <errno.h>
#include <stdlib.h>

#define NAME "/tmp/gfortran-xxxxxxx"

typedef struct shared_res
{
  void *ptr;
  size_t size;
  int fd;
  int status;
} shared_res_t;

shared_res_t
get_shared_resource (size_t size, bool master)
{
  shared_res_t res = {0};
  int fd = -1;
  if (master)
    fd = open (NAME, O_CREAT | O_RDWR,  S_IRUSR | S_IWUSR);
  else
    while ((fd = open (NAME, O_RDWR,  S_IRUSR | S_IWUSR)) < 0 
	   && errno == ENOENT) usleep (100000);

  if (fd == -1)
    {
      perror ("get_shared_resource: open failed");
      goto fatal_err;
    }
  else
    printf ("get_shared_resource: fd = %d\n", fd);

  if (master)
    if (ftruncate (fd, size) < 0)
      {
	perror ("get_shared_resource: ftruncate failed");
        goto fatal_err;
      }

  void *p1 = mmap (NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : mmap failed");
      goto fatal_err;
    }
  res.fd = fd;
  res.ptr = p1;
  res.size = size;
  res.status = 0;
  return res;

fatal_err:
  if (fd >= 0)
    close (fd);
  shm_unlink (NAME);
  exit (EXIT_FAILURE);  
}

void
resize_shared_resource (shared_res_t *r, size_t new_size)
{
  if (r->status != 0)
    {
      perror ("resize_shared_resource : bad shared resource status");
      return;
    }

  if (munmap (r->ptr, r->size) < 0 )
    perror ("resize_shared_resource : unmap failed");

  if (ftruncate (r->fd, new_size) < 0)
    {
      perror ("resize_shared_resource: ftruncate failed");
      return;
    }

  /* remap this */
  void *p1 = mmap (NULL, new_size, PROT_READ | PROT_WRITE, MAP_SHARED, r->fd, 0);
  if (p1 == MAP_FAILED)
    perror ("resize_shared_resource : mmap failed");
  else
    {
      r->ptr = p1;
      r->size = new_size;
    }
}

void release_shared_resource (shared_res_t r)
{
  if (munmap (r.ptr, r.size) < 0 )
    {
      perror ("release_shared_resource : unmap failed");
    }
  close (r.fd);
  unlink (NAME);
}

#if SENDER

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("sender page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, true);
  printf ("sender resource pointer = 0x%p\n", shared.ptr);
  int *ip = shared.ptr;
  *ip = 42;

  while (*ip == 42)
    usleep (100000);

  printf ("sender saw %d, resizing now\n", *ip);
  resize_shared_resource (&shared, 2 * pagesize);
  printf ("sender resized resource pointer = 0x%p\n", shared.ptr);

  ip = shared.ptr;
  ip += (pagesize/sizeof(int));
  printf ("sender sees %d\n", *ip);

  release_shared_resource (shared);
  return 0;
}

#else

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("client page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, false);
  printf ("client resource pointer = 0x%p\n", shared.ptr);

  int *ip = shared.ptr;

  while (*ip != 42)
    usleep (100000);

  printf ("client saw %d, resizing now\n", *ip);
  resize_shared_resource (&shared, 2 * pagesize);
  printf ("client resized resource pointer = 0x%p\n", shared.ptr);

  ip = shared.ptr;
  ip[(pagesize/sizeof(int))] = 6174;
  *ip = -42;

  release_shared_resource (shared);
  return 0;
}

#endif


[-- Attachment #5: Type: text/plain, Size: 3 bytes --]