Update on shared memory coarrays

Update on shared memory coarrays

[Nicolas König]

Hello world,

here is an update for the coarray_native branch.

TL;DR: New stuff and bugfixes, need help with testsuite integration.

Toon's test case has provided the opportunity to fix numerous
bugs; it now runs correctly.  It can be found at
http://moene.org/~toon/random-weather .

Here's a list of the major changes since the last email:

* According to popular demand, the library has been renamed
   cas_shared, and any references to "native coarrays" have been expunged
   (except for the branch name, which we cannot change). the directory under
   libgfortran has been moved to cas_shared and the prefix in the front 
end and
   library has been changed to cas (for coarray_shared).  The option is also
   called -fcoarray=shared now.

* Use of pthread_barrier_t and associated functions has been removed, so 
there
   is a chance that the library compiles on MacOS now.

* STAT and ERRMSG have been implemented.

So, generally, the branch seems to be in a usable state now. Major
missing features at the moment are:

* allocatable components in coarrays
* teams
* gcc testsuite integration and testcases

Since I'd like to get the branch merged as quickly as possible, any
help with the testsuite would be greatly appreciated.

So, if you could give it a spin (especially to test if there are
platforms where it does not bootstrap), that would be great.

   Nicolas

Re: Update on shared memory coarrays

[Iain Sandoe]

Hi Nicolas,

Nicolas König <koenigni@student.ethz.ch> wrote:
> On 05/01/2021 15:58, Iain Sandoe wrote:
>> Hi Nicolas,
>> Nicolas König <koenigni@student.ethz.ch> wrote:
>>>>>>>

>>>> This will work from macOS 10.7+ (Darwin11+)
>>>
>>> Finally some good news.
>>>
>>>> earlier OS versions fail with
>>>> pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED)failed:  
>>>> Undefined error: 0
>>>> (the compile succeeds, but some functionality is missing).
>>>
>>> Shared mutexes are absolutely necessary, so we probably will never  
>>> support macOS<10.7, but -given it was released 9 years ago- I don't  
>>> feel too bad about that.
>> I don’t feel too bad about it either (not bad enough to divert scarce  
>> effort into finding an alterate solution, currently anyway)***
>> However, there needs to be a strategy for handling it ( e.g. don’t try  
>> to build the library / disable the functionality ) on platforms without  
>> required support.
>
> The test I send you should probably be added as a configure test,

FWIW, the reason that the test reports "failed: Undefined error: 0” is  
becuase the pthread functions return the error code, but do not set errno  
(at least apparently not on Linux or Darwin) .. if one does
   if ((tmp =  x) != 0) … errno = tmp; perror … then the output shows “INVAL” as expected.

A run test isn’t really ideal for configury in the general case (it can’t  
work for cross-compilers - much less for a canadian cross).

> though I haven't heard anything about the branch causing build errors.

It won’t cause a build error - the pthread_mutexattr_setpshared() (and  
corresponding cond one) are present and conform to SUSv2 on the earlier  
systems.  That means that they accept PTHREAD_PROCESS_PRIVATE, but not  
PTHREAD_PROCESS_SHARED.

===

In this case, since the [pthreads] functionality is a property of the  
installed libc [and that’s fixed for each OS version], we can just test the  
target OS version and use that as a gate.   Actually, that’s a pretty cheap  
config test.

If it [that pthreads function] is absolutely required for the [coarrays]  
library to be of any use, it’s better to disable the library build (and the  
functionity completely) that on systems without support - otherwise expect  
PRs from confused users…

cheers
Iain

Re: Update on shared memory coarrays

[Nicolas König]

Hi Iain

On 05/01/2021 15:58, Iain Sandoe wrote:
> Hi Nicolas,
> 
> Nicolas König <koenigni@student.ethz.ch> wrote:
> 
>>>>>> It also doesn't bring my hopes up that other things like 
>>>>>> pthread_{mutex,cond}attr_setpshared() will work.
>>>>> Darwin is Posix SUSv3 (actually, AFAICT the majority of the certified
>>>>> platforms are; there only seems to be one certified to UNIX7).
>>>>> - so pthreads stuff that’s mandated in SUSv3 is definitely present 
>>>>> on Darwin.
>>>>> (in practice some of the optional stuff is also present - you can 
>>>>> look at
>>>>> what libstdc++ does to get an idea of what works in a 
>>>>> cross-platform manner)
>>>>
>>>> Process shared mutexes/condition variables are optional :) Attached 
>>>> is a test case that should check whether the two work.
>>>> <shared_cond_test.c>
>>> This will work from macOS 10.7+ (Darwin11+)
>>
>> Finally some good news.
>>
>>> earlier OS versions fail with
>>> pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED)failed: 
>>> Undefined error: 0
>>> (the compile succeeds, but some functionality is missing).
>>
>> Shared mutexes are absolutely necessary, so we probably will never 
>> support macOS<10.7, but -given it was released 9 years ago- I don't 
>> feel too bad about that.
> 
> I don’t feel too bad about it either (not bad enough to divert scarce 
> effort into finding an alterate solution, currently anyway)***
> 
> However, there needs to be a strategy for handling it ( e.g. don’t try 
> to build the library / disable the functionality ) on platforms without 
> required support.

The test I send you should probably be added as a configure test, though 
I haven't heard anything about the branch causing build errors.

> 
> GCC is built on macOS back to 10.4 [I even test it over that range] 
>   (and macports and homebrew have support for many OSS packages) - it’s 
> completely reasonable that these (much) older platforms have reduced or 
> absent functionality - but GCC should continue to build there, and work 
> at least as much as it does now ;)

Absolutely agreed. I'm trying to be as careful as possible not to break 
bootstrap anywhere :D

   Nicolas

> 
> cheers
> Iain
> 
> *** note that libstdc++ has encountered most of the pthreads issues and 
> might already have solutions.
> 

Re: Update on shared memory coarrays

[Nicolas König]

Hi Iain,

On 05/01/2021 15:28, Iain Sandoe wrote:
> Hi Nicolas,
> 
> Nicolas König <koenigni@student.ethz.ch> wrote:
>>
>> On 05/01/2021 14:11, Iain Sandoe wrote:
>>> Hi folks,
>>> Nicolas König <koenigni@student.ethz.ch> wrote:
>>>> On 04/01/2021 03:34, Iain Sandoe wrote:
>>>>> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
>>>>>> Am 03.01.21 um 16:42 schrieb Iain Sandoe:
>>>>> It seems shm is not a favoured IPC mechanism for Darwin (it’s 
>>>>> possibly  disallowed for GUI apps, according to search results - I 
>>>>> didn’t check  that)
>>>>>>> (we can file a radar, and maybe it gets fixed at some point - 
>>>>>>> but  that’s no help to all the existing system versions)
>>>>>>
>>>>>> Makes sense to file a bug report with Apple.  And yes, we'll have to
>>>>>> deal with this some other way.
>>>>> [for command line apps] Either mechanism for accessing shm works;
>>>>> [1] shm_open / fd / mmap / munmap
>>>>> [2] shmget / shmat / shmdt <= allocation is rounded up to the 
>>>>> nearest  page boundary.
>>>>> Both have the constraint [on Darwin] that the size of the 
>>>>> allocation  cannot be increased after it is set (by ftruncate() [1] 
>>>>> or shmget() [2]).
>>>>> It can be decreased (by [2] only), but it’s not clear if that 
>>>>> actually  has any physical effect.
>>>>
>>>> That's a problem, since coarrays can be allocated at runtime, and 
>>>> we  can't possibly precompute their size.
>>> OK.
>>>> To get coarrays to work, we would probably need to create new 
>>>> shared  memory objects (with predictable names, so they can be 
>>>> opened from other  images), but that would be a significant time 
>>>> investment.
>>> How does it work between different images with shared memory unless the
>>> name to shm_open () is predictable between images?
>>
>> Currently, the images are launched by one "master"-process, that also 
>> handles all the management. There, we can get some shared memory via 
>> mmap(MAP_ANON | MAP_SHARED), so the master could communicate the base 
>> name of the image. At the moment, it just opens a shared_memory object 
>> and passes the file descriptor to its children.
> 
> Ah, I see - so the name can in that case be arbitrary
> OTOH, what if one runs multiple Fortran codes (as the same user)?
> (some simple hashing with the the master’s PID would fix that, I guess)

That's already being done :) At the moment, the name is something like 
gfortran_shared_$PID_$ID, with ID some random number that is incremented 
if opening it fails because of O_EXCL. As you said, if we need to use 
shmget, then we could probably hash that string.

> 
>>>> It also doesn't bring my hopes up that other things like 
>>>> pthread_{mutex,cond}attr_setpshared() will work.
>>> Darwin is Posix SUSv3 (actually, AFAICT the majority of the certified
>>> platforms are; there only seems to be one certified to UNIX7).
>>> - so pthreads stuff that’s mandated in SUSv3 is definitely present on 
>>> Darwin.
>>> (in practice some of the optional stuff is also present - you can 
>>> look at
>>> what libstdc++ does to get an idea of what works in a cross-platform 
>>> manner)
>>
>> Process shared mutexes/condition variables are optional :) Attached is 
>> a test case that should check whether the two work.
> 
>> <shared_cond_test.c>
> 
> This will work from macOS 10.7+ (Darwin11+)

Finally some good news.

> 
> earlier OS versions fail with
> pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED)failed: 
> Undefined error: 0
> 
> (the compile succeeds, but some functionality is missing).

Shared mutexes are absolutely necessary, so we probably will never 
support macOS<10.7, but -given it was released 9 years ago- I don't feel 
too bad about that.

   Nicolas

> 
> Iain
> 

Re: Update on shared memory coarrays

[Nicolas König]

[-- Attachment #1: Type: text/plain, Size: 5556 bytes --]

Hi Iain,

On 05/01/2021 14:11, Iain Sandoe wrote:
> Hi folks,
> 
> Nicolas König <koenigni@student.ethz.ch> wrote:
>> On 04/01/2021 03:34, Iain Sandoe wrote:
>>> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
>>>> Am 03.01.21 um 16:42 schrieb Iain Sandoe:
> 
>>> It seems shm is not a favoured IPC mechanism for Darwin (it’s possibly  
>>> disallowed for GUI apps, according to search results - I didn’t check  
>>> that)
>>>>> (we can file a radar, and maybe it gets fixed at some point - but  
>>>>> that’s no help to all the existing system versions)
>>>>
>>>> Makes sense to file a bug report with Apple.  And yes, we'll have to
>>>> deal with this some other way.
>>> [for command line apps] Either mechanism for accessing shm works;
>>> [1] shm_open / fd / mmap / munmap
>>> [2] shmget / shmat / shmdt <= allocation is rounded up to the nearest  
>>> page boundary.
>>> Both have the constraint [on Darwin] that the size of the allocation  
>>> cannot be increased after it is set (by ftruncate() [1] or shmget() [2]).
>>> It can be decreased (by [2] only), but it’s not clear if that actually  
>>> has any physical effect.
>>
>> That's a problem, since coarrays can be allocated at runtime, and we  
>> can't possibly precompute their size.
> 
> OK.
> 
>> To get coarrays to work, we would probably need to create new shared  
>> memory objects (with predictable names, so they can be opened from other  
>> images), but that would be a significant time investment.
> 
> How does it work between different images with shared memory unless the
> name to shm_open () is predictable between images?

Currently, the images are launched by one "master"-process, that also 
handles all the management. There, we can get some shared memory via 
mmap(MAP_ANON | MAP_SHARED), so the master could communicate the base 
name of the image. At the moment, it just opens a shared_memory object 
and passes the file descriptor to its children.

> 
> (there has to be some shared key of some form, right?)
> 
>> It also doesn't bring my hopes up that other things like  
>> pthread_{mutex,cond}attr_setpshared() will work.
> 
> Darwin is Posix SUSv3 (actually, AFAICT the majority of the certified
> platforms are; there only seems to be one certified to UNIX7).
> 
> - so pthreads stuff that’s mandated in SUSv3 is definitely present on 
> Darwin.
> 
> (in practice some of the optional stuff is also present - you can look at
> what libstdc++ does to get an idea of what works in a cross-platform manner)

Process shared mutexes/condition variables are optional :) Attached is a 
test case that should check whether the two work.

> 
>> I think it's probably better to focus the attention on getting it to work  
>> on Linux/BSD.
> 
> Well, I can understand making progress in that way - but it’s a bit of a
> slippery slope towards an “x86 Linux only” compiler (if we keep taking that
> route until a significant portion of the useful features don’t work
> elsewhere).

As a person that uses a ppc64le machine as his work station, I can 
completely understand that concern.

> 
> Of course, this particular case is not so exclusive [it appears from
> previous mails that Darwin is the only one with an issue], but as a general
> rule it would be good to pick an impl that works on at least the regularly
> used/tested targets.
> 
>>> I guess that one ought to consider that a change in allocation could  
>>> trigger a copy on any system (since there might not be contiguous memory  
>>> pages available for an arbitrary increase) - so a strategy which either  
>>> allocates separae fragments for each need - or waits until the size is  
>>> known before allocation is likely to perform better on average,
>>
>> There's no need to have the continuous virtual memory backed by  
>> continuous physical memory, so nothing here should trigger a relocation  
>> of the actual physical memory. There are just some entries in the page  
>> table pointing to the same pages.
> 
> I guess if everything is done on page boundaries & sizes, that can work -
> providing one doesn’t mind the base pointer changing across resizes.
> 
> ====
> 
> Update:
> 
> three codes attached.
> 
> 1. Posix shm implementation; works fine but is not resizeable.
> 
> 2. Darwin shm implementation : works fine but is only resizable downwards
> and it’s not clear if the resources are actually released.

At the moment, the shared memory is never unmapped but instead handed to 
an allocator, so the ability to make it smaller is optional. Either of 
these should work, if we create more than one. They should also give 
better perf than 3.

> 
> 3. mmapped shared file : This works [on Darwin at least] and is resizable.
> I would assume that, in reality, all accesses would be to the buffer cache
> [unless something actually forces the cache to be flushed] and therefore
> that the performance ought to be similar to shm.

This would probably be the fastest way to get some implementation to 
work, but the page fault/context switch at each access to a coarray hurts.

> 
> Worst case, one could point /tmp at a ramdisk ..
> (I tend to do that anyway, I don’t want my sshds getting hammered by
> toolchain builds).
> 
> Thoughts?

I think the best way to proceed is to introduce a new level of 
abstraction over the shared memory object that makes it easier to swap 
an alternative implementation depending on the system (I think the 
current shared_memory.c/h still has too much other logic in it, but I'm 
not sure).

   Nicolas

> Iain
> 
> 
> 
> 

[-- Attachment #2: shared_cond_test.c --]
[-- Type: text/x-csrc, Size: 971 bytes --]

#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h>

#define ERR_CHK(x) \
  do { \
	if (x) { \
	  perror(#x "failed"); \
	  exit(1); \
	} \
  } while (0)

int
main(int argc, char **argv) {
  pthread_condattr_t cattr;
  pthread_mutexattr_t mattr;
  pthread_mutex_t *m;
  pthread_cond_t *c;
  void *mem;

  ERR_CHK((mem = mmap(0, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANON, -1, 0)) == MAP_FAILED);

  m = mem;
  c = mem + 0x800;

  ERR_CHK(pthread_mutexattr_init(&mattr));
  ERR_CHK(pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED));
  ERR_CHK(pthread_mutex_init(m, &mattr));
  ERR_CHK(pthread_mutexattr_destroy(&mattr));

  ERR_CHK(pthread_condattr_init(&cattr));
  ERR_CHK(pthread_condattr_setpshared(&cattr, PTHREAD_PROCESS_SHARED));
  ERR_CHK(pthread_cond_init(c, &cattr));
  ERR_CHK(pthread_condattr_destroy(&cattr));

  printf("Success\n");

  return 0;
}

Re: Update on shared memory coarrays

[Iain Sandoe]

Hi Nicolas,

Nicolas König <koenigni@student.ethz.ch> wrote:
>
> On 05/01/2021 14:11, Iain Sandoe wrote:
>> Hi folks,
>> Nicolas König <koenigni@student.ethz.ch> wrote:
>>> On 04/01/2021 03:34, Iain Sandoe wrote:
>>>> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
>>>>> Am 03.01.21 um 16:42 schrieb Iain Sandoe:
>>>> It seems shm is not a favoured IPC mechanism for Darwin (it’s  
>>>> possibly  disallowed for GUI apps, according to search results - I  
>>>> didn’t check  that)
>>>>>> (we can file a radar, and maybe it gets fixed at some point - but   
>>>>>> that’s no help to all the existing system versions)
>>>>>
>>>>> Makes sense to file a bug report with Apple.  And yes, we'll have to
>>>>> deal with this some other way.
>>>> [for command line apps] Either mechanism for accessing shm works;
>>>> [1] shm_open / fd / mmap / munmap
>>>> [2] shmget / shmat / shmdt <= allocation is rounded up to the nearest   
>>>> page boundary.
>>>> Both have the constraint [on Darwin] that the size of the allocation   
>>>> cannot be increased after it is set (by ftruncate() [1] or shmget()  
>>>> [2]).
>>>> It can be decreased (by [2] only), but it’s not clear if that  
>>>> actually  has any physical effect.
>>>
>>> That's a problem, since coarrays can be allocated at runtime, and we   
>>> can't possibly precompute their size.
>> OK.
>>> To get coarrays to work, we would probably need to create new shared   
>>> memory objects (with predictable names, so they can be opened from  
>>> other  images), but that would be a significant time investment.
>> How does it work between different images with shared memory unless the
>> name to shm_open () is predictable between images?
>
> Currently, the images are launched by one "master"-process, that also  
> handles all the management. There, we can get some shared memory via  
> mmap(MAP_ANON | MAP_SHARED), so the master could communicate the base  
> name of the image. At the moment, it just opens a shared_memory object  
> and passes the file descriptor to its children.

Ah, I see - so the name can in that case be arbitrary
OTOH, what if one runs multiple Fortran codes (as the same user)?
(some simple hashing with the the master’s PID would fix that, I guess)

>>> It also doesn't bring my hopes up that other things like  
>>> pthread_{mutex,cond}attr_setpshared() will work.
>> Darwin is Posix SUSv3 (actually, AFAICT the majority of the certified
>> platforms are; there only seems to be one certified to UNIX7).
>> - so pthreads stuff that’s mandated in SUSv3 is definitely present on  
>> Darwin.
>> (in practice some of the optional stuff is also present - you can look at
>> what libstdc++ does to get an idea of what works in a cross-platform  
>> manner)
>
> Process shared mutexes/condition variables are optional :) Attached is a  
> test case that should check whether the two work.

> <shared_cond_test.c>

This will work from macOS 10.7+ (Darwin11+)

earlier OS versions fail with
pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED)failed:  
Undefined error: 0

(the compile succeeds, but some functionality is missing).

Iain

Re: Update on shared memory coarrays

[Iain Sandoe]

Thomas Koenig <tkoenig@netcologne.de> wrote:

> Am 05.01.21 um 14:11 schrieb Iain Sandoe:
>
>> Well, I can understand making progress in that way - but it’s a bit of a  
>> slippery slope towards an “x86 Linux only” compiler
>
> It's a slippery slope towards an "POWER, ARM + x86 only" compiler for
> FreeBSD, NetBSD, OpenBSD, AIX, OpenSolaris and, yes, Linux.

Yeah, as I said in my following sentence:

"Of course, this particular case is not so exclusive [it appears from
previous mails that Darwin is the only one with an issue], but as a general
rule it would be good to pick an impl that works on at least the regularly
used/tested targets.”

anyway, not my place to make policy - but I have put forward a working impl,

Iain

Re: Update on shared memory coarrays

[Thomas Koenig]

Am 05.01.21 um 14:11 schrieb Iain Sandoe:

> Well, I can understand making progress in that way - but it’s a bit of a 
> slippery slope towards an “x86 Linux only” compiler

It's a slippery slope towards an "POWER, ARM + x86 only" compiler for
FreeBSD, NetBSD, OpenBSD, AIX, OpenSolaris and, yes, Linux.

All these systems seem to work.

Best regards

	Thomas

Re: Update on shared memory coarrays

[Iain Sandoe]

Iain Sandoe <iain@sandoe.co.uk> wrote:


> Update:
>
> three codes attached.
>
> 1. Posix shm implementation; works fine but is not resizeable.
>
> 2. Darwin shm implementation : works fine but is only resizable downwards
> and it’s not clear if the resources are actually released.
>
> 3. mmapped shared file : This works [on Darwin at least] and is resizable.
> I would assume that, in reality, all accesses would be to the buffer cache
> [unless something actually forces the cache to be flushed] and therefore
> that the performance ought to be similar to shm.
>
> Worst case, one could point /tmp at a ramdisk ..
> (I tend to do that anyway, I don’t want my sshds getting hammered by
> toolchain builds).
>
> <shm-posix.c><shm-darwin.c><ram-file.c>


… I omitted  :

gcc <file> -DSENDER -o A
gcc <file> -o B

(output for the working case, ram-file)

A &
sender page size = 4096
get_shared_resource: fd = 3
sender resource pointer = 0x0x103d1b000

B &
client page size = 4096
get_shared_resource: fd = 3
client resource pointer = 0x0x10e3a7000
client saw 42, resizing now
client resized resource pointer = 0x0x10e3a7000

sender saw -42, resizing now
sender resized resource pointer = 0x0x103d1b000
sender sees 6174

Re: Update on shared memory coarrays

[Iain Sandoe]

[-- Attachment #1: Type: text/plain, Size: 4112 bytes --]

Hi folks,

Nicolas König <koenigni@student.ethz.ch> wrote:
> On 04/01/2021 03:34, Iain Sandoe wrote:
>> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
>>> Am 03.01.21 um 16:42 schrieb Iain Sandoe:

>> It seems shm is not a favoured IPC mechanism for Darwin (it’s possibly  
>> disallowed for GUI apps, according to search results - I didn’t check  
>> that)
>>>> (we can file a radar, and maybe it gets fixed at some point - but  
>>>> that’s no help to all the existing system versions)
>>>
>>> Makes sense to file a bug report with Apple.  And yes, we'll have to
>>> deal with this some other way.
>> [for command line apps] Either mechanism for accessing shm works;
>> [1] shm_open / fd / mmap / munmap
>> [2] shmget / shmat / shmdt <= allocation is rounded up to the nearest  
>> page boundary.
>> Both have the constraint [on Darwin] that the size of the allocation  
>> cannot be increased after it is set (by ftruncate() [1] or shmget() [2]).
>> It can be decreased (by [2] only), but it’s not clear if that actually  
>> has any physical effect.
>
> That's a problem, since coarrays can be allocated at runtime, and we  
> can't possibly precompute their size.

OK.

> To get coarrays to work, we would probably need to create new shared  
> memory objects (with predictable names, so they can be opened from other  
> images), but that would be a significant time investment.

How does it work between different images with shared memory unless the  
name to shm_open () is predictable between images?

(there has to be some shared key of some form, right?)

> It also doesn't bring my hopes up that other things like  
> pthread_{mutex,cond}attr_setpshared() will work.

Darwin is Posix SUSv3 (actually, AFAICT the majority of the certified  
platforms are; there only seems to be one certified to UNIX7).

- so pthreads stuff that’s mandated in SUSv3 is definitely present on Darwin.

(in practice some of the optional stuff is also present - you can look at  
what libstdc++ does to get an idea of what works in a cross-platform manner)

> I think it's probably better to focus the attention on getting it to work  
> on Linux/BSD.

Well, I can understand making progress in that way - but it’s a bit of a  
slippery slope towards an “x86 Linux only” compiler (if we keep taking that  
route until a significant portion of the useful features don’t work  
elsewhere).

Of course, this particular case is not so exclusive [it appears from  
previous mails that Darwin is the only one with an issue], but as a general  
rule it would be good to pick an impl that works on at least the regularly  
used/tested targets.

>> I guess that one ought to consider that a change in allocation could  
>> trigger a copy on any system (since there might not be contiguous memory  
>> pages available for an arbitrary increase) - so a strategy which either  
>> allocates separae fragments for each need - or waits until the size is  
>> known before allocation is likely to perform better on average,
>
> There's no need to have the continuous virtual memory backed by  
> continuous physical memory, so nothing here should trigger a relocation  
> of the actual physical memory. There are just some entries in the page  
> table pointing to the same pages.

I guess if everything is done on page boundaries & sizes, that can work -  
providing one doesn’t mind the base pointer changing across resizes.

====

Update:

three codes attached.

1. Posix shm implementation; works fine but is not resizeable.

2. Darwin shm implementation : works fine but is only resizable downwards  
and it’s not clear if the resources are actually released.

3. mmapped shared file : This works [on Darwin at least] and is resizable.   
I would assume that, in reality, all accesses would be to the buffer cache  
[unless something actually forces the cache to be flushed] and therefore  
that the performance ought to be similar to shm.

Worst case, one could point /tmp at a ramdisk ..
(I tend to do that anyway, I don’t want my sshds getting hammered by  
toolchain builds).

Thoughts?
Iain


[-- Attachment #2: shm-posix.c --]
[-- Type: application/octet-stream, Size: 3252 bytes --]

/* shared memory IPC using Posix shm_open/ftruncate/mmap.
   The Darwin implementation does not appear to allow a shared memory segment
   to be resized once the initial size is set.
*/

#include <stdbool.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/shm.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>

#define NAME "/tmp/random_namexxx"

typedef struct shared_res
{
  void *ptr;
  size_t size;
  int fd;
  int status;
} shared_res_t;

shared_res_t
get_shared_resource (size_t size, bool master)
{
  shared_res_t res = {0};

  int fd;
  if (master)
    /* Create the segment.  */
   fd = shm_open (NAME, O_CREAT | O_RDWR,  S_IRUSR | S_IWUSR);
  else
    /* Wait for the segment to be created.  */
    while ((fd = shm_open (NAME, O_RDWR,  S_IRUSR | S_IWUSR)) < 0 
	   && errno == ENOENT) usleep (100000);

  if (fd < 0)
    {
      perror ("get_shared_resource: open failed");
      goto fatal_err;
    }
  else
    printf ("get_shared_resource: fd = %d\n", fd);

  if (master)
    {
      if (ftruncate (fd, size) < 0)
	{
	  perror ("get_shared_resource: ftruncate failed");
  	  goto fatal_err;
	}
    }

  void *p1 = mmap (NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : mmap failed");
      goto fatal_err;
    }
  res.fd = fd;
  res.ptr = p1;
  res.size = size;
  res.status = 0;
  return res;

fatal_err:
  if (fd >= 0)
    close (fd);
  shm_unlink (NAME);
  exit (EXIT_FAILURE);  
}

void
resize_shared_resource (shared_res_t *r, size_t new_size)
{
  if (r->status != 0)
    {
      perror ("resize_shared_resource : bad shared resource status");
      goto fatal_err;
    }

  if (ftruncate (r->fd, new_size) < 0)
    {
      perror ("resize_shared_resource: ftruncate failed");
      goto fatal_err;
    }

  /* TODO: remap this if we succeeded ... but we do not*/
  return;

fatal_err:
  if (r->fd >= 0)
    close (r->fd);
  shm_unlink (NAME);
  exit (EXIT_FAILURE);  
}

void release_shared_resource (shared_res_t r)
{
  if (munmap (r.ptr, r.size) < 0 )
    {
      perror ("release_shared_resource : unmap failed");
    }
  shm_unlink (NAME);
  close (r.fd);
}

#if SENDER

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("sender page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, true);
  printf ("sender shared = 0x%p\n", shared.ptr);
  int *ip = shared.ptr;
  *ip = 42;

  while (*ip == 42)
    usleep (100000);

  /* Any attempt to re-size this (increase or decrease) fails.
  resize_shared_resource (&shared, pagesize * 2);
  */
  release_shared_resource (shared);
  return 0;
}

#else

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("client page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, false);
  printf ("client shared = 0x%p\n", shared.ptr);

  int *ip = shared.ptr;

  while (*ip != 42)
    usleep (100000);

  *ip = 6174;

  release_shared_resource (shared);
  return 0;
}

#endif

[-- Attachment #3: shm-darwin.c --]
[-- Type: application/octet-stream, Size: 3892 bytes --]

/* shared memory IPC using shmget/shmat/shmdt.
   There doesn't appear to be any mechanism to re-size such a segment
   (although re-getting with a smaller size produces no error, it doesn't
    make the segment size contained in the status any smaller).
*/

#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/shm.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <unistd.h>

typedef struct shared_res
{
  void *ptr;
  size_t size;
  int smid;
  int status;
  key_t savedKey;
} shared_res_t;

shared_res_t
get_shared_resource (size_t size)
{
  shared_res_t res = {0};

  key_t myKey = ftok ("/dev/null", 6174);
  if (myKey < 0)
    {
      perror ("ftok failed");
      exit (EXIT_FAILURE);
    }

  int smid = shmget (myKey, size, IPC_CREAT | SHM_R | SHM_W);
  if (smid < 0)
    {
      perror ("shmget failed");
      exit (EXIT_FAILURE);
    }
  printf ("smid = %d\n", smid);

  void *p1 = shmat (smid, NULL, SHM_R | SHM_W);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : shmat failed");
      exit (EXIT_FAILURE);
    }
  res.smid = smid;
  res.ptr = p1;
  res.size = size;
  res.status = 0;
  res.savedKey = myKey;
  return res;
}

void
resize_shared_resource (shared_res_t *r, size_t new_size)
{
  struct shmid_ds sb;
  int res = shmctl(r->smid, IPC_STAT, &sb);
  if (res < 0)
    {
      perror ("shmctl (1) failed");
      exit (EXIT_FAILURE);
    }
  printf ("size in shm = %zu\n", sb.shm_segsz);

  /* An attempt to get the same segment with a different size fails.
     detach and start again.  */
  if (shmdt (r->ptr) < 0 )
    {
      perror ("resize_shared_resource : shmdt failed");
    }

  int smid1 = shmget (r->savedKey, new_size, IPC_CREAT | SHM_R | SHM_W);
  if (smid1 < 0)
    {
      perror ("shmget (2) failed");
    }
  printf ("smid1 = %d\n", smid1);
  void *p1 = shmat (smid1, NULL, SHM_R | SHM_W);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : shmat failed");
      exit (EXIT_FAILURE);
    }
  r->smid = smid1;
  r->ptr = p1;
  r->size = new_size;  
}

void release_shared_resource (shared_res_t r)
{
  /* detach.  */
  if (shmdt (r.ptr) < 0 )
    perror ("release_shared_resource : shmdt failed");

  struct shmid_ds sb;
  int res = shmctl(r.smid, IPC_STAT, &sb);
  if (res < 0)
    {
      perror ("release_shared_resource : shmctl failed");
      return;
    }

  printf ("release_shared_resource : size in shm = %zu\n", sb.shm_segsz);
  /* If this is the last use, then remove the shm segment.  */
  if (sb.shm_nattch == 0)
    {
      res = shmctl(r.smid, IPC_RMID, &sb);
      if (res < 0)
	{
	  perror ("release_shared_resource : shmctl (IPC_RMID) failed");
	}
    } 
}

#if SENDER

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("sender page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize * 2);
  printf ("sender shared = 0x%p\n", shared.ptr);
  int *ip = shared.ptr;
  *ip = 42;

  while (*ip == 42)
    usleep (100000);

  /* A resize to a smaller segment proceeds without error, but the actual
     reported size from the shm segment is not changed.  */

  printf ("sender attempting resize to %zu\n", pagesize);
  resize_shared_resource (&shared, pagesize);
  printf ("sender releasing\n");
  release_shared_resource (shared);
  return 0;
}

#else

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("client page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize * 2);
  printf ("client shared = 0x%p\n", shared.ptr);

  int *ip = shared.ptr;

  while (*ip != 42)
    usleep (100000);

  *ip = 6174;

  printf ("client releasing\n");
  release_shared_resource (shared);
  return 0;
}

#endif

[-- Attachment #4: ram-file.c --]
[-- Type: application/octet-stream, Size: 3545 bytes --]

/* shared resource IPC using mmapped file.
   This allows resizing.
*/

#include <stdbool.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/mman.h>
#include <errno.h>
#include <stdlib.h>

#define NAME "/tmp/gfortran-xxxxxxx"

typedef struct shared_res
{
  void *ptr;
  size_t size;
  int fd;
  int status;
} shared_res_t;

shared_res_t
get_shared_resource (size_t size, bool master)
{
  shared_res_t res = {0};
  int fd = -1;
  if (master)
    fd = open (NAME, O_CREAT | O_RDWR,  S_IRUSR | S_IWUSR);
  else
    while ((fd = open (NAME, O_RDWR,  S_IRUSR | S_IWUSR)) < 0 
	   && errno == ENOENT) usleep (100000);

  if (fd == -1)
    {
      perror ("get_shared_resource: open failed");
      goto fatal_err;
    }
  else
    printf ("get_shared_resource: fd = %d\n", fd);

  if (master)
    if (ftruncate (fd, size) < 0)
      {
	perror ("get_shared_resource: ftruncate failed");
        goto fatal_err;
      }

  void *p1 = mmap (NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  if (p1 == MAP_FAILED)
    {
      perror ("get_shared_resource : mmap failed");
      goto fatal_err;
    }
  res.fd = fd;
  res.ptr = p1;
  res.size = size;
  res.status = 0;
  return res;

fatal_err:
  if (fd >= 0)
    close (fd);
  shm_unlink (NAME);
  exit (EXIT_FAILURE);  
}

void
resize_shared_resource (shared_res_t *r, size_t new_size)
{
  if (r->status != 0)
    {
      perror ("resize_shared_resource : bad shared resource status");
      return;
    }

  if (munmap (r->ptr, r->size) < 0 )
    perror ("resize_shared_resource : unmap failed");

  if (ftruncate (r->fd, new_size) < 0)
    {
      perror ("resize_shared_resource: ftruncate failed");
      return;
    }

  /* remap this */
  void *p1 = mmap (NULL, new_size, PROT_READ | PROT_WRITE, MAP_SHARED, r->fd, 0);
  if (p1 == MAP_FAILED)
    perror ("resize_shared_resource : mmap failed");
  else
    {
      r->ptr = p1;
      r->size = new_size;
    }
}

void release_shared_resource (shared_res_t r)
{
  if (munmap (r.ptr, r.size) < 0 )
    {
      perror ("release_shared_resource : unmap failed");
    }
  close (r.fd);
  unlink (NAME);
}

#if SENDER

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("sender page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, true);
  printf ("sender resource pointer = 0x%p\n", shared.ptr);
  int *ip = shared.ptr;
  *ip = 42;

  while (*ip == 42)
    usleep (100000);

  printf ("sender saw %d, resizing now\n", *ip);
  resize_shared_resource (&shared, 2 * pagesize);
  printf ("sender resized resource pointer = 0x%p\n", shared.ptr);

  ip = shared.ptr;
  ip += (pagesize/sizeof(int));
  printf ("sender sees %d\n", *ip);

  release_shared_resource (shared);
  return 0;
}

#else

int main()
{
  size_t pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }
  printf ("client page size = %ld\n", pagesize);

  shared_res_t shared = get_shared_resource (pagesize, false);
  printf ("client resource pointer = 0x%p\n", shared.ptr);

  int *ip = shared.ptr;

  while (*ip != 42)
    usleep (100000);

  printf ("client saw %d, resizing now\n", *ip);
  resize_shared_resource (&shared, 2 * pagesize);
  printf ("client resized resource pointer = 0x%p\n", shared.ptr);

  ip = shared.ptr;
  ip[(pagesize/sizeof(int))] = 6174;
  *ip = -42;

  release_shared_resource (shared);
  return 0;
}

#endif


[-- Attachment #5: Type: text/plain, Size: 3 bytes --]

Re: Update on shared memory coarrays

[Nicolas König]

Hello Iain,

First of all, thank you for investigating this so thoroughly.

On 04/01/2021 03:34, Iain Sandoe wrote:
> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
>> Am 03.01.21 um 16:42 schrieb Iain Sandoe:
>>> [SHM] Shared Memory Objects
>>> The functionality described is optional. The functionality described 
>>> is also an extension to the ISO C standard.
>>> … so support is optional and a failure in partial support is not an 
>>> actual bug in compliance …
>>
>> It's a bug anyway :-)
> 
> certainly, it’s inconvenient having different behaviout from other 
> platforms

Yes, completely agree.

> 
> It seems shm is not a favoured IPC mechanism for Darwin (it’s possibly 
> disallowed for GUI apps, according to search results - I didn’t check that)
> 
>>> (we can file a radar, and maybe it gets fixed at some point - but 
>>> that’s no help to all the existing system versions)
>>
>> Makes sense to file a bug report with Apple.  And yes, we'll have to
>> deal with this some other way.
> 
> [for command line apps] Either mechanism for accessing shm works;
> 
> [1] shm_open / fd / mmap / munmap
> 
> [2] shmget / shmat / shmdt <= allocation is rounded up to the nearest 
> page boundary.
> 
> Both have the constraint [on Darwin] that the size of the allocation 
> cannot be increased after it is set (by ftruncate() [1] or shmget() [2]).
> It can be decreased (by [2] only), but it’s not clear if that actually 
> has any physical effect.

That's a problem, since coarrays can be allocated at runtime, and we 
can't possibly precompute their size.

To get coarrays to work, we would probably need to create new shared 
memory objects (with predictable names, so they can be opened from other 
images), but that would be a significant time investment. It also 
doesn't bring my hopes up that other things like 
pthread_{mutex,cond}attr_setpshared() will work.

I think it's probably better to focus the attention on getting it to 
work on Linux/BSD.

> 
> I guess that one ought to consider that a change in allocation could 
> trigger a copy on any system (since there might not be contiguous memory 
> pages available for an arbitrary increase) - so a strategy which either 
> allocates separae fragments for each need - or waits until the size is 
> known before allocation is likely to perform better on average,

There's no need to have the continuous virtual memory backed by 
continuous physical memory, so nothing here should trigger a relocation 
of the actual physical memory. There are just some entries in the page 
table pointing to the same pages.

   Nicolas

> 
> Iain
> 

Re: Update on shared memory coarrays

[Iain Sandoe]

Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
> Am 03.01.21 um 16:42 schrieb Iain Sandoe:
>> [SHM] Shared Memory Objects
>> The functionality described is optional. The functionality described is  
>> also an extension to the ISO C standard.
>> … so support is optional and a failure in partial support is not an  
>> actual bug in compliance …
>
> It's a bug anyway :-)

certainly, it’s inconvenient having different behaviout from other platforms.

It seems shm is not a favoured IPC mechanism for Darwin (it’s possibly  
disallowed for GUI apps, according to search results - I didn’t check that)

>> (we can file a radar, and maybe it gets fixed at some point - but that’s  
>> no help to all the existing system versions)
>
> Makes sense to file a bug report with Apple.  And yes, we'll have to
> deal with this some other way.

[for command line apps] Either mechanism for accessing shm works;

[1] shm_open / fd / mmap / munmap

[2] shmget / shmat / shmdt <= allocation is rounded up to the nearest page  
boundary.

Both have the constraint [on Darwin] that the size of the allocation cannot  
be increased after it is set (by ftruncate() [1] or shmget() [2]).
It can be decreased (by [2] only), but it’s not clear if that actually has  
any physical effect.

I guess that one ought to consider that a change in allocation could  
trigger a copy on any system (since there might not be contiguous memory  
pages available for an arbitrary increase) - so a strategy which either  
allocates separae fragments for each need - or waits until the size is  
known before allocation is likely to perform better on average,

Iain

Re: Update on shared memory coarrays

[Thomas Koenig]

Am 03.01.21 um 16:42 schrieb Iain Sandoe:
> [SHM] Shared Memory Objects
> The functionality described is optional. The functionality described is 
> also an extension to the ISO C standard.
> 
> … so support is optional and a failure in partial support is not an 
> actual bug in compliance …

It's a bug anyway :-)

> (we can file a radar, and maybe it gets fixed at some point - but that’s 
> no help to all the existing system versions)

Makes sense to file a bug report with Apple.  And yes, we'll have to
deal with this some other way.

Best regards

	Thomas

Re: Update on shared memory coarrays

[Iain Sandoe]

(continuing the monologue)

Iain Sandoe <iain@sandoe.co.uk> wrote:

> Iain Sandoe <iain@sandoe.co.uk> wrote:
>
>> Iain Sandoe <iain@sandoe.co.uk> wrote:
>>
>>> Iain Sandoe <iain@sandoe.co.uk> wrote:
>

> However shm_open interface *does* appear to allow arbitrary sized  
> segments [at least not reporting an error to the first call to  
> ftruncate()] - but to deny a resize once created.
>
> seems more background reading will be needed …

So… Posix 1003 says...

[SHM]  If fildes refers to a shared memory object, ftruncate() shall set  
the size of the shared memory object to length.

… which means it "should work as observed on Linux "…

… BUT…. Posix 1003 also says

[SHM] Shared Memory Objects
The functionality described is optional. The functionality described is  
also an extension to the ISO C standard.

… so support is optional and a failure in partial support is not an actual  
bug in compliance …

(we can file a radar, and maybe it gets fixed at some point - but that’s no  
help to all the existing system versions)

=====

whether there’s some way to coerce the alternate interfaces into working,  
remains to be seen
Iain

Re: Update on shared memory coarrays

[Iain Sandoe]

Iain Sandoe <iain@sandoe.co.uk> wrote:

> Iain Sandoe <iain@sandoe.co.uk> wrote:
>
>> Iain Sandoe <iain@sandoe.co.uk> wrote:
>>
>>> Nicolas König <koenigni@student.ethz.ch> wrote:
>>> [IIUC what you mean by double-mapping] Allocating the same thing twice  
>>> seems likely to be an accident waiting to happen :) ..
>>
>> So the intent is:
>>
>> shm_allocate
>>
>> size the shm
>>
>> mmap the shm
>>
>> resize the shm

>> will poke some more if I have a chance.
>
> There is an alternate shm interface on Darwin that allows manipulation of  
> the shm object more directly (and mapping/unmapping etc).
>
> It’s not clear if that mechanism will allow the shm object to be resized,  
> it doesn’t say in the man page that one cannot - but it doesn’t say one  
> can either.
>
> If I have a chance, I’ll try to recode the testcase to use that interface  
> and see if it works.

Well .. (with this interface) I can create a half-page shm segment, and  
then expand that to a whole page

.. but it doesn’t seem to allow > 1 page in the segment.  That limitation  
is not spelled out in the man pages (might still be some mistake in my  
usage).

AFAICT it rounds up to a page, which means that the second request can be  
honoured without a reallocation.

It seems that the total shared memory supported by the (configured values)  
in the Darwin kernel might be quite small anyway :

$ sysctl -a |grep shm
kern.sysv.shmmax: 4194304
kern.sysv.shmmin: 1
kern.sysv.shmmni: 32
kern.sysv.shmseg: 8
kern.sysv.shmall: 1024

However shm_open interface *does* appear to allow arbitrary sized segments  
[at least not reporting an error to the first call to ftruncate()] - but to  
deny a resize once created.

seems more background reading will be needed …
Iain

Re: Update on shared memory coarrays

[Iain Sandoe]

Iain Sandoe <iain@sandoe.co.uk> wrote:

> Iain Sandoe <iain@sandoe.co.uk> wrote:
>
>> Nicolas König <koenigni@student.ethz.ch> wrote:
>> [IIUC what you mean by double-mapping] Allocating the same thing twice  
>> seems likely to be an accident waiting to happen :) ..
>
> So the intent is:
>
> shm_allocate
>
> size the shm
>
> mmap the shm
>
> resize the shm

Actually, it’s not the second mapping that’s failing - it’s the attempt to  
resize the shm by using ftruncate() a second time with a larger value.

A quick test says Linux seems to accept the resize, Darwin doesn’t (maybe  
this is a Darwin bug - not sure if the ability to resize via this mechanism  
is a Posix SUSv3 requirement - need to read the Posix info).

[aside: I tried various other permutations, such as duping the fd and then  
attempting to modify the dup, also using fcntl with F_PREALLOCATE]

Anyway, the attempt to resize appears to render the fd invalid, and then  
any subsequent use of it fails (which is what the second mmap is reporting).

Without doing the resize, a second mmap succeeds (AFAICT).

> will poke some more if I have a chance.

There is an alternate shm interface on Darwin that allows manipulation of  
the shm object more directly (and mapping/unmapping etc).

It’s not clear if that mechanism will allow the shm object to be resized,  
it doesn’t say in the man page that one cannot - but it doesn’t say one can  
either.

If I have a chance, I’ll try to recode the testcase to use that interface  
and see if it works.

OTOH, perhaps you already have an idea about how to side-step the issue -  
as noted before it seems possible that the resize would involve a  
reallocation (I note that the sucessful second mmap on Linux does have a  
different starting address - but that doesn’t prove it was copied, of  
course).

Iain

Re: Update on shared memory coarrays

[Nicolas König]

Hello Iain,

On 02/01/2021 21:55, Iain Sandoe wrote:
> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
> 
>>> map_memory image 0 size 1000 offset 0 pagesize 1000
>>> map_memory image 0 size 2000 offset 0 pagesize 1000
> 
> pagesize 1000?

Pretty sure that's hex :)

> 
> Darwin’s pagesize is 4096 for PPC/X86 and 16384 for Arm64(aarch64).
> 
>>> mmap failed: Invalid argument
>>
>> That means that a rewrite of shared_memory.c and shared_memory.h
>> is required.
>>
>> I checked the attached program on every Unixoid system I
>> could lay my hands on (Linux, AIX, OpenBSD, NetBSD, FreeBSD,
>> OpenSolaris), and it passed everywhere.  Seems like MacOS
>> is the only one where this particular idiom does not work.
> 
> Perhaps I’m missing what the idiom is here :)

At the moment, we are double-mapping pages in the shared_memory. This 
was on the list of things that should be changed, but according to 
Thomas's tests, you can't do this on MacOS, so it just got a whole lot 
more important.

   Nicolas

> 
> (a bit tied up with other things at the moment so mainly watching from
>   the peanut gallery for now)
> 
> Iain
> 

Re: Update on shared memory coarrays

[Iain Sandoe]

Iain Sandoe <iain@sandoe.co.uk> wrote:

> Nicolas König <koenigni@student.ethz.ch> wrote:
>
>
> [IIUC what you mean by double-mapping] Allocating the same thing twice  
> seems likely to be an accident waiting to happen :) ..

So the intent is:

shm_allocate

size the shm

mmap the shm

resize the shm

mmap the resized (without releasing the first mapping)

(with the intent to preserve the contents of the original mapped entities,  
I guess?)

hmm… ok (there do seem to be some potential pitfalls there, depending on  
whether the system needs to reallocate / move the object in  response to  
the re-sizing).

will poke some more if I have a chance.
Iain

Re: Update on shared memory coarrays

[Iain Sandoe]

Nicolas König <koenigni@student.ethz.ch> wrote:

> On 02/01/2021 21:55, Iain Sandoe wrote:
>> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
>>>> map_memory image 0 size 1000 offset 0 pagesize 1000
>>>> map_memory image 0 size 2000 offset 0 pagesize 1000
>> pagesize 1000?
>
> Pretty sure that's hex :)

I wondered, but worth asking …

>>> I checked the attached program on every Unixoid system I
>>> could lay my hands on (Linux, AIX, OpenBSD, NetBSD, FreeBSD,
>>> OpenSolaris), and it passed everywhere.  Seems like MacOS
>>> is the only one where this particular idiom does not work.
>> Perhaps I’m missing what the idiom is here :)
>
> At the moment, we are double-mapping pages in the shared_memory. This was  
> on the list of things that should be changed, but according to Thomas's  
> tests, you can't do this on MacOS, so it just got a whole lot more  
> important.
>
>  Nicolas
>
>> (a bit tied up with other things at the moment so mainly watching from
>>  the peanut gallery for now)

^ this remains true .. but ..

… can you separate what is being attempted into a small test-case (perhaps  
that’s already been done?) that I could poke at without having to build a  
branch?

[IIUC what you mean by double-mapping] Allocating the same thing twice  
seems likely to be an accident waiting to happen :) ..

Iain

Re: Update on shared memory coarrays

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]

Am 02.01.21 um 21:55 schrieb Iain Sandoe:
> Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:
> 
>>> map_memory image 0 size 1000 offset 0 pagesize 1000
>>> map_memory image 0 size 2000 offset 0 pagesize 1000
> 
> pagesize 1000?

Output is in hex (easy to miss, as I didn't put the 0x into
the printf statements :-)

> Darwin’s pagesize is 4096 for PPC/X86 and 16384 for Arm64(aarch64).
> 
>>> mmap failed: Invalid argument
>>
>> That means that a rewrite of shared_memory.c and shared_memory.h
>> is required.
>>
>> I checked the attached program on every Unixoid system I
>> could lay my hands on (Linux, AIX, OpenBSD, NetBSD, FreeBSD,
>> OpenSolaris), and it passed everywhere.  Seems like MacOS
>> is the only one where this particular idiom does not work.
> 
> Perhaps I’m missing what the idiom is here :)

The idiom is remapping a shared memory region a second time into
memory, starting from the same offset (zero in this case).  This is
currently used to make the shared memory allocator more simple, and is
something that, from Dominique's debugging, does not work on MacOS.

It is something that can be worked around (says the person who didn't
write the allocatator in the first place :-) but it's additional work.

By the way, the original test case I posted to the list had an error,
here is an updated one.

Best regards

	Thomas

[-- Attachment #2: shm_open.c --]
[-- Type: text/x-csrc, Size: 1185 bytes --]

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>

#define NAME "/random_namexxx"

int main()
{

  int fd;
  long pagesize;
  void *p1, *p2;
  volatile int *i1, *i2;
  size_t size1, size2;
  off_t offset;

  pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }

  fd = shm_open (NAME, O_CREAT | O_EXCL | O_RDWR,  S_IRUSR | S_IWUSR);
  if (fd == -1)
    {
      perror ("shm_open failed");
      exit (EXIT_FAILURE);
    }
  shm_unlink (NAME);

  offset = 0;
  size1 = pagesize;
  ftruncate (fd, size1);
  p1 = mmap (NULL, size1, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
  if (p1 == MAP_FAILED)
    {
      perror ("mmap 1 failed");
      exit (EXIT_FAILURE);
    }
  printf ("p1 = %p\n", p1);
  i1 = p1;
  *i1 = 42;
  size2 = 2 * size1;
  ftruncate (fd, size2);
  p2 = mmap (NULL, size2, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
  if (p2 == MAP_FAILED)
    {
      perror ("mmap 1 failed");
      exit (EXIT_FAILURE);
    }
  printf ("p2 = %p\n", p2);
  i2 = p2;
  printf ("%d\n", *i2);
  return 0;
}

Re: Update on shared memory coarrays

[Iain Sandoe]

Thomas Koenig via Fortran <fortran@gcc.gnu.org> wrote:

>> map_memory image 0 size 1000 offset 0 pagesize 1000
>> map_memory image 0 size 2000 offset 0 pagesize 1000

pagesize 1000?

Darwin’s pagesize is 4096 for PPC/X86 and 16384 for Arm64(aarch64).

>> mmap failed: Invalid argument
>
> That means that a rewrite of shared_memory.c and shared_memory.h
> is required.
>
> I checked the attached program on every Unixoid system I
> could lay my hands on (Linux, AIX, OpenBSD, NetBSD, FreeBSD,
> OpenSolaris), and it passed everywhere.  Seems like MacOS
> is the only one where this particular idiom does not work.

Perhaps I’m missing what the idiom is here :)

(a bit tied up with other things at the moment so mainly watching from
  the peanut gallery for now)

Iain

Re: Update on shared memory coarrays

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]


Hi Dominique,

> map_memory image 0 size 1000 offset 0 pagesize 1000
> map_memory image 0 size 2000 offset 0 pagesize 1000
> mmap failed: Invalid argument

That means that a rewrite of shared_memory.c and shared_memory.h
is required.

I checked the attached program on every Unixoid system I
could lay my hands on (Linux, AIX, OpenBSD, NetBSD, FreeBSD,
OpenSolaris), and it passed everywhere.  Seems like MacOS
is the only one where this particular idiom does not work.

Oh well...

Best regards

	Thomas

[-- Attachment #2: shm_open.c --]
[-- Type: text/x-csrc, Size: 1185 bytes --]

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>

#define NAME "/random_namexxx"

int main()
{

  int fd;
  long pagesize;
  void *p1, *p2;
  volatile int *i1, *i2;
  size_t size1, size2;
  off_t offset;

  pagesize = sysconf (_SC_PAGE_SIZE);
  if (pagesize == -1)
    {
      perror ("sysconf failed");
      exit (EXIT_FAILURE);
    }

  fd = shm_open (NAME, O_CREAT | O_EXCL | O_RDWR,  S_IRUSR | S_IWUSR);
  if (fd == -1)
    {
      perror ("shm_open failed");
      exit (EXIT_FAILURE);
    }
  shm_unlink (NAME);

  offset = 0;
  size1 = pagesize;
  p1 = mmap (NULL, size1, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
  if (p1 == MAP_FAILED)
    {
      perror ("mmap 1 failed");
      exit (EXIT_FAILURE);
    }
  printf ("p1 = %p\n", p1);
  ftruncate (fd, size1);
  i1 = p1;
  *i1 = 42;
  size2 = 2 * size1;
  p2 = mmap (NULL, size2, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
  if (p2 == MAP_FAILED)
    {
      perror ("mmap 1 failed");
      exit (EXIT_FAILURE);
    }
  printf ("p2 = %p\n", p2);
  i2 = p2;
  ftruncate (fd, size2);
  printf ("%d\n", *i2);
  return 0;
}

Re: Update on shared memory coarrays

[dhumieres.dominique]

Le 2021-01-01 17:16, Thomas Koenig a écrit :
> Hi Dominique,
> 
> Happy new year to you, too, and to everybody else!
> 
>> Now the test fails with
>> 
>> mmap failed: Invalid argument
>> 
>> AFAICT this happens on the second call to map_memory.
> 
> A quick question.
> 
> Could you try the following diagnostic patch
> 
> diff --git a/libgfortran/caf_shared/shared_memory.c
> b/libgfortran/caf_shared/shared_memory.c
> index b64e40a3ded..03990550f59 100644
> --- a/libgfortran/caf_shared/shared_memory.c
> +++ b/libgfortran/caf_shared/shared_memory.c
> @@ -79,6 +79,11 @@ map_memory (int fd, size_t size, off_t offset)
>  {
>    void *ret
>        = mmap (NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 
> offset);
> +  fprintf (stderr, "map_memory image %d size %lx offset %lx pagesize 
> %lx\n",
> +          this_image.image_num + 1, (unsigned long) size,
> +          (unsigned long) offset,
> +          (unsigned long) pagesize);
> +
>    if (ret == MAP_FAILED)
>      {program main
>   real, dimension(:), allocatable :: a[:]
>   real, dimension(:), allocatable :: b[:]
>   allocate (a(1000)[*])
>   allocate (b(2000)[3:*])
> end program main
>        perror ("mmap failed");
> 
> 
> to make sure that mmap is indeed getting the right parameters?
> 
> On my Linux box, the output looks like
> 
> map_memory image 0 size 1000 offset 0 pagesize 1000
> map_memory image 4 size 5000 offset 0 pagesize 1000
> map_memory image 2 size 5000 offset 0 pagesize 1000
> map_memory image 1 size 5000 offset 0 pagesize 1000
> map_memory image 3 size 5000 offset 0 pagesize 1000
> map_memory image 3 size d000 offset 0 pagesize 1000
> map_memory image 2 size d000 offset 0 pagesize 1000
> map_memory image 1 size d000 offset 0 pagesize 1000
> map_memory image 4 size d000 offset 0 pagesize 1000
> 
> 
> for the following little program:
> 
> program main
>   real, dimension(:), allocatable :: a[:]
>   real, dimension(:), allocatable :: b[:]
>   allocate (a(1000)[*])
>   allocate (b(2000)[3:*])
> end program main
> 
> Best regards
> 
> 	Thomas

I get

map_memory image 0 size 1000 offset 0 pagesize 1000
map_memory image 0 size 2000 offset 0 pagesize 1000
mmap failed: Invalid argument

Best regards

Dominique

Re: Update on shared memory coarrays

[Thomas Koenig]

Hi Dominique,

Happy new year to you, too, and to everybody else!

> Now the test fails with
> 
> mmap failed: Invalid argument
> 
> AFAICT this happens on the second call to map_memory.

A quick question.

Could you try the following diagnostic patch

diff --git a/libgfortran/caf_shared/shared_memory.c 
b/libgfortran/caf_shared/shared_memory.c
index b64e40a3ded..03990550f59 100644
--- a/libgfortran/caf_shared/shared_memory.c
+++ b/libgfortran/caf_shared/shared_memory.c
@@ -79,6 +79,11 @@ map_memory (int fd, size_t size, off_t offset)
  {
    void *ret
        = mmap (NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset);
+  fprintf (stderr, "map_memory image %d size %lx offset %lx pagesize 
%lx\n",
+          this_image.image_num + 1, (unsigned long) size,
+          (unsigned long) offset,
+          (unsigned long) pagesize);
+
    if (ret == MAP_FAILED)
      {program main
   real, dimension(:), allocatable :: a[:]
   real, dimension(:), allocatable :: b[:]
   allocate (a(1000)[*])
   allocate (b(2000)[3:*])
end program main
        perror ("mmap failed");


to make sure that mmap is indeed getting the right parameters?

On my Linux box, the output looks like

map_memory image 0 size 1000 offset 0 pagesize 1000
map_memory image 4 size 5000 offset 0 pagesize 1000
map_memory image 2 size 5000 offset 0 pagesize 1000
map_memory image 1 size 5000 offset 0 pagesize 1000
map_memory image 3 size 5000 offset 0 pagesize 1000
map_memory image 3 size d000 offset 0 pagesize 1000
map_memory image 2 size d000 offset 0 pagesize 1000
map_memory image 1 size d000 offset 0 pagesize 1000
map_memory image 4 size d000 offset 0 pagesize 1000


for the following little program:

program main
   real, dimension(:), allocatable :: a[:]
   real, dimension(:), allocatable :: b[:]
   allocate (a(1000)[*])
   allocate (b(2000)[3:*])
end program main

Best regards

	Thomas

Re: Update on shared memory coarrays

[dhumieres.dominique]

Le 2020-12-27 18:10, Nicolas König a écrit :
> Hi Dominique,
> 
> Thank you for the report. We figured out that for some reason, shared
> memory objects can't have names longer than PSHMMEMLEN (set to 31)
> characters.
> 
> We just committed a fix that shortens the name on Darwin and checks
> whether it actually failed with EEXIST or whether it was some other
> error. It would be great of you could try again the newest version :)
> 
> Best
> 
>   Nicolas
> 

Thanks for looking at the problem.
Now the test fails with

mmap failed: Invalid argument

AFAICT this happens on the second call to map_memory.

Happy new year!

Dominique

Re: Update on shared memory coarrays

[Nicolas König]

Hi Dominique,

Thank you for the report. We figured out that for some reason, shared 
memory objects can't have names longer than PSHMMEMLEN (set to 31) 
characters.

We just committed a fix that shortens the name on Darwin and checks 
whether it actually failed with EEXIST or whether it was some other 
error. It would be great of you could try again the newest version :)

Best

   Nicolas

On 23/12/2020 18:25, dhumieres.dominique@free.fr wrote:
> Le 2020-12-22 16:10, Thomas Koenig a écrit :
>> Hi Dominique,
>>
>>> Indeed! I can try to do some debugging under supervision.
>>> Could you please tell me where to start?
>>
>> Thanks for your offer.
>>
>> The first thing to do would be to see where it hangs.  Generating
>> a syscall trace (by whatever tool is used on MacOS, dtrace? ktrace?)
>> would be a first step.
>>
>> A next step would be to compile a program with debug info and
>> run it under a debugger, then see where it hangs.
>>
>> I'm not sure - do you use gdb on your system?  If you do,
>> the following lines in your .gdbinit
>>
>> define better_mp_debug
>>   set detach-on-fork off
>>   set schedule-multiple on
>>   set follow-fork-mode parent
>>   set non-stop on
>>   set target-async on
>>   set print symbol-loading off
>> end
>>
>> will give you a macro "better_mp_debug" which will ease debugging
>> of forked processes enormously.
>>
>> My lack of experience with MacOS extends to which tools to use
>> for debugging, unfortunately.
>>
>> Best regards
>>
>>     Thomas
> 
> Running the hello test under lldb with breakpoints at ensure_initialization
> and get_shmem_fd gives a first stop at __gfortran_cas_master,
> a second one at __gfortrani_ensure_initialization, and a third one at
> __gfortrani_ensure_initialization. Then if I adavance step by step
> the program seems to loop in
> 
>    do
>      {
>        snprintf (buffer, sizeof (buffer), MEMOBJ_NAME "_%u_%d",
>                  (unsigned int)getpid (), id++);
>        fd = shm_open (buffer, O_RDWR | O_CREAT | O_EXCL, S_IRUSR | 
> S_IWUSR);
>      }
>    while (fd == -1);
> 
> fd is always -1 and
> 
> (lldb) p errno
> (void *) $1 = 0x000000000000003f
> 
> I don't see anything wrong with shm_open.
> In the config.log I see both
> 
> ac_cv_func_shm_open=yes
> ac_cv_func_shm_unlink=yes
> 
> and
> 
> ac_cv_lib_pthread_pthread_create=yes
> ac_cv_lib_rt_shm_open=no
> 
> What should I do next?
> 
> TIA
> 
> Domonique

Re: Update on shared memory coarrays

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]


Hi Paul,

after a longish debugging session with Nicolas, we finally
discovered what was wrong.  Apart from what you pointed out,
I had also managed to take the address of a NULL pointer,
which led to the strange fails every now and then.

Here is the patch as committed.

Best regards

	Thomas

Fix errors introduced by last commit.

gcc/fortran/ChangeLog:

	* trans-array.c (gfc_conv_array_ref): Before discarding offset,
	check if this is a coarray and we are building shared coarrays.
	* trans-decl.c (gfc_build_builtin_function_decls): Clear up
	types for cas_coarray_allocate.
	(gfc_trans_shared_coarray): Pass NULL_TREEs to status and
	errmsg.

libgfortran/ChangeLog:

	* caf_shared/util.h (CAS_DEBUG_PR): New macro.

gcc/testsuite/ChangeLog:

	* gfortran.dg/caf-shared/cas.exp: Add -g to debug flags to allow
	better backtrace.

[-- Attachment #2: p.diff --]
[-- Type: text/x-patch, Size: 3481 bytes --]

diff --git a/gcc/fortran/trans-array.c b/gcc/fortran/trans-array.c
index 35afff5845e..58aaa5f781d 100644
--- a/gcc/fortran/trans-array.c
+++ b/gcc/fortran/trans-array.c
@@ -3935,7 +3935,7 @@ gfc_conv_array_ref (gfc_se * se, gfc_array_ref * ar, gfc_expr *expr,
 
   /* Early return - only taken for ALLOCATED for shared coarrays.
      FIXME - this could probably be done more elegantly.  */
-  if (se->address_only)
+  if (flag_coarray == GFC_FCOARRAY_SHARED && ar->codimen && se->address_only)
     {
       se->expr = build_array_ref (se->expr, build_int_cst (TREE_TYPE (offset), 0),
 				  decl, se->class_vptr);
diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c
index f3526db7ea6..ab2725ca6f1 100644
--- a/gcc/fortran/trans-decl.c
+++ b/gcc/fortran/trans-decl.c
@@ -3736,8 +3736,11 @@ gfc_build_intrinsic_function_decls (void)
 void
 gfc_build_builtin_function_decls (void)
 {
+  tree gfc_int4_type_node = gfc_get_int_type (4);
+  tree gfc_pint4_type_node = build_pointer_type (gfc_int4_type_node);
   tree gfc_int8_type_node = gfc_get_int_type (8);
   tree pint_type = build_pointer_type (integer_type_node);
+  tree pchar1_type_node = gfc_get_pchar_type (1);
 
   gfor_fndecl_stop_numeric = gfc_build_library_function_decl (
 	get_identifier (PREFIX("stop_numeric")),
@@ -4123,8 +4126,8 @@ gfc_build_builtin_function_decls (void)
 	 size_type_node,	/* elem_size.  */
 	 integer_type_node,	/* corank.  */
 	 integer_type_node,	/* alloc_type.  */
-	 pvoid_type_node,	/* stat.  */
-	 pvoid_type_node,	/* errmsg.  */
+	 gfc_pint4_type_node,	/* stat.  */
+	 pchar1_type_node,	/* errmsg.  */
 	 gfc_charlen_type_node, /* errmsg_len.  */
 	 NULL_TREE);
       gfor_fndecl_cas_coarray_free = gfc_build_library_function_decl_with_spec (
@@ -4697,8 +4700,8 @@ gfc_trans_shared_coarray (stmtblock_t * init, stmtblock_t *cleanup, gfc_symbol *
 			   NULL_TREE, true, NULL, &element_size);
       elem_size = size_in_bytes (gfc_get_element_type (TREE_TYPE(decl)));
       gfc_allocate_shared_coarray (init, decl, elem_size, sym->as->rank,
-				   sym->as->corank, alloc_type, null_pointer_node,
-				   null_pointer_node,
+				   sym->as->corank, alloc_type,
+				   NULL_TREE, NULL_TREE,
 				   build_int_cst (gfc_charlen_type_node, 0),
 				   false);
       gfc_conv_descriptor_offset_set (init, decl, offset);
diff --git a/gcc/testsuite/gfortran.dg/caf-shared/cas.exp b/gcc/testsuite/gfortran.dg/caf-shared/cas.exp
index 86e6b97090b..8f73bf24b88 100644
--- a/gcc/testsuite/gfortran.dg/caf-shared/cas.exp
+++ b/gcc/testsuite/gfortran.dg/caf-shared/cas.exp
@@ -47,7 +47,7 @@ foreach test [lsort [glob -nocomplain $srcdir/$subdir/*.\[fF\]{,90,95,03,08} ]]
     if ![runtest_file_p $runtests $test] then {
 	continue
     }
-    set option_list [list { -O2 } ]
+    set option_list [list { -g -O2 } ]
 
     set nshort [file tail [file dirname $test]]/[file tail $test]
     list-module-names $test
diff --git a/libgfortran/caf_shared/util.h b/libgfortran/caf_shared/util.h
index fa9e158b38f..6ae37780fb4 100644
--- a/libgfortran/caf_shared/util.h
+++ b/libgfortran/caf_shared/util.h
@@ -28,6 +28,10 @@ see the files COPYING3 and COPYING.RUNTIME respectively.  If not, see
 #include <pthread.h>
 #include <limits.h>
 #include <assert.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#define CAS_DEBUG_PR(str, ...) dprintf(2, "Image %d (pid %ld):\t" str "\n", this_image.image_num, (unsigned long) getpid(), ##__VA_ARGS__)
 
 #define PTR_BITS (CHAR_BIT*sizeof(void *))
 

Re: Update on shared memory coarrays

[Thomas Koenig]

Hi Paul,

> The patch applied cleanly, as might be expected, but resulted in the 
> following failures on FC3/x86_64 (plus assumed_type_9.f90):
> 
> FAIL: gfortran.dg/caf-shared/atomic_1.f90 -pthread -fcoarray=shared  -O2 
>   -lcaf_shared -lrt execution test
> FAIL: gfortran.dg/caf-shared/image_index_2.f90 -pthread -fcoarray=shared 
>   -O2  -lcaf_shared -lrt execution test
> FAIL: gfortran.dg/caf-shared/this_image_2.f90 -pthread -fcoarray=shared 
>   -O2  -lcaf_shared -lrt execution test

I don't think I will ever figure out the testsuite :-(

I see the same failures you do on "make check-fortran", but I do not
shee the caf-shared failures with

$ make check-fortran RUNTESTFLAGS="cas.exp=*"

which I had done before. So, making a regular "make check-fortran"
is the way to go.

So, that particular way (which was rather hackish, I admit)
was not the right one.

Best regards

	Thomas

Re: Update on shared memory coarrays

[Paul Richard Thomas]

Hi Thomas and Nicolas,

The patch applied cleanly, as might be expected, but resulted in the
following failures on FC3/x86_64 (plus assumed_type_9.f90):

FAIL: gfortran.dg/caf-shared/atomic_1.f90 -pthread -fcoarray=shared  -O2
 -lcaf_shared -lrt execution test
FAIL: gfortran.dg/caf-shared/image_index_2.f90 -pthread -fcoarray=shared
 -O2  -lcaf_shared -lrt execution test
FAIL: gfortran.dg/caf-shared/this_image_2.f90 -pthread -fcoarray=shared
 -O2  -lcaf_shared -lrt execution test
.........
FAIL: gfortran.dg/alloc_comp_deep_copy_1.f03   -O0  execution test
FAIL: gfortran.dg/alloc_comp_deep_copy_1.f03   -O1  execution test
FAIL: gfortran.dg/alloc_comp_deep_copy_1.f03   -O2  execution test
FAIL: gfortran.dg/alloc_comp_deep_copy_1.f03   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/alloc_comp_deep_copy_1.f03   -O3 -g  execution test
FAIL: gfortran.dg/alloc_comp_deep_copy_1.f03   -Os  execution test

This latter is failing at  stop 66, stop 83, stop 153, stop 170. In these
cases,
if (.not. allocated(o2%vec(2)%ai))
is translated to:
if ((*(struct innert[0:] * restrict) o2.vec.data)[0].ai == 0B) , which is
manifestly incorrect.

Oddly, at the next line
if (o2%vec(2)%ai /= 7)
is correctly translated to:
if (*(*(struct innert[0:] * restrict) o2.vec.data)[o2.vec.offset + 2].ai !=
7)

This is caused by the following chunk in the patch:
@@ -3934,6 +3933,15 @@ gfc_conv_array_ref (gfc_se * se, gfc_array_ref * ar,
gfc_expr *expr,
  decl = NULL_TREE;
     }

+  /* Early return - only taken for ALLOCATED for shared coarrays.
+     FIXME - this could probably be done more elegantly.  */
+  if (se->address_only)                               // CHANGING TO if
(se->address_only && ar->codimen) FIXES PROBLEM
+    {
+      se->expr = build_array_ref (se->expr, build_int_cst (TREE_TYPE
(offset), 0),
+  decl, se->class_vptr);
+      return;
+    }
+
   se->expr = build_array_ref (se->expr, offset, decl, se->class_vptr);
 }

I don't know if adding ar->codimen was what was intended. The origin of
this is in:
trans-intrinsic.c(gfc_conv_allocated), where the condition needs to check
for array references:
      if (arg1->expr->rank == 0)
{
 /* Allocatable scalar.  */
 arg1se.want_pointer = 1;
 arg1se.address_only = 1;


Cheers

Paul


On Wed, 23 Dec 2020 at 10:47, Thomas Koenig <tkoenig@netcologne.de> wrote:

> Hi everybody,
>
> after Nicolas' offlineOK, I have now committed the attached patch
> to the branch.
>
> This should fix the test cases which are already committed
> (hopefully...).  And I did add a FIXME for something that should
> be cleaned up at a later date.
>
> Regarding the failures that Steve reported, we'll try to reproduce
> this using -m32 bits later - right now, Christmas is approching fast :-)
>
> Best regards
>
>         Thomas
>
>
> Add offset to allocatable shared coarrays.
>
> This adds the calculation of the offset for allocatable coarrays,
> which was missing before, and fixes the resulting fallout for
> ALLOCATED.  Additionally, it prepares the way for STAT and ERRMSG
> for ALLOCATE of coarrays, but that still needs changes to
> gfc_trans_allocate.
>
> gcc/fortran/ChangeLog:
>
>         * trans-array.c (gfc_conv_array_ref): If se->address_only is set,
>         throw away all the offset calculation.
>         (gfc_allocate_shared_coarray): Add arguments stat, errmsg and
>         errlen to call to allocate.  Calculate offset for allocatable
>         coarrays.
>         (gfc_array_allocate): Adjust call to gfc_allocate_shared_coarray.
>         * trans-array.h (gfc_allocate_shared_coarray): Change prototype
>         of cas_coarray_alloc.
>         * trans-decl.c (gfc_build_builtin_function_decls): Adjust
>         cas_coarray_alloc to changed prototypes.
>         (gfc_trans_shared_coarray): Adjust call to
> gfc_allocate_shared_coarray.
>         * trans-intrinsic.c (gfc_conv_allocated): Set address_only on se.
>         * trans.h: Add flag address_only to gfc_se.
>
> libgfortran/ChangeLog:
>
>         * caf_shared/wrapper.c (cas_coarray_alloc): Add status, error and
>         errmsg arguments and their checking.
>


-- 
"If you can't explain it simply, you don't understand it well enough" -
Albert Einstein

Re: Update on shared memory coarrays

[dhumieres.dominique]

Le 2020-12-22 16:10, Thomas Koenig a écrit :
> Hi Dominique,
> 
>> Indeed! I can try to do some debugging under supervision.
>> Could you please tell me where to start?
> 
> Thanks for your offer.
> 
> The first thing to do would be to see where it hangs.  Generating
> a syscall trace (by whatever tool is used on MacOS, dtrace? ktrace?)
> would be a first step.
> 
> A next step would be to compile a program with debug info and
> run it under a debugger, then see where it hangs.
> 
> I'm not sure - do you use gdb on your system?  If you do,
> the following lines in your .gdbinit
> 
> define better_mp_debug
>   set detach-on-fork off
>   set schedule-multiple on
>   set follow-fork-mode parent
>   set non-stop on
>   set target-async on
>   set print symbol-loading off
> end
> 
> will give you a macro "better_mp_debug" which will ease debugging
> of forked processes enormously.
> 
> My lack of experience with MacOS extends to which tools to use
> for debugging, unfortunately.
> 
> Best regards
> 
> 	Thomas

Running the hello test under lldb with breakpoints at 
ensure_initialization
and get_shmem_fd gives a first stop at __gfortran_cas_master,
a second one at __gfortrani_ensure_initialization, and a third one at
__gfortrani_ensure_initialization. Then if I adavance step by step
the program seems to loop in

   do
     {
       snprintf (buffer, sizeof (buffer), MEMOBJ_NAME "_%u_%d",
                 (unsigned int)getpid (), id++);
       fd = shm_open (buffer, O_RDWR | O_CREAT | O_EXCL, S_IRUSR | 
S_IWUSR);
     }
   while (fd == -1);

fd is always -1 and

(lldb) p errno
(void *) $1 = 0x000000000000003f

I don't see anything wrong with shm_open.
In the config.log I see both

ac_cv_func_shm_open=yes
ac_cv_func_shm_unlink=yes

and

ac_cv_lib_pthread_pthread_create=yes
ac_cv_lib_rt_shm_open=no

What should I do next?

TIA

Domonique

Re: Update on shared memory coarrays

[Nicolas König]

Hi Paul,

On 23/12/2020 10:38, Paul Richard Thomas wrote:
> Hi Thomas,
> 
> OK on the failures - I'll look out for the patch.

Thomas suggested that patches for the branch should be posted to the 
mailing list. At the moment, I usually approve them via backchannels :D

> 
> Is there a list somewhere of what has not been implemented but is 
> implemented in the existing coarrays library? Also, what is not 
> implemented from the standard?

Thomas and I added a todo-list in the readme file in the 
libgfortran/caf_shared directory. We're currently looking at allocate 
with stat; the compiler is emitting some rather strange code there. It 
resets, depending on the uninitialized value of stat, some of the array 
bounds.

Library side, the next task will be to add the library functions needed 
for implementing MOVE_ALLOC.

   Nicolas

> 
> Many thanks for your stupendous effort on this >
> I am going to tackle the fixes for my two recent patches that have hit 
> the buffers PR83118 (patch as of last night) and PR98022 (patch that 
> fixes it but removes all the work that I have put into it :-( ). I'll 
> post patches today.
> 
> Paul
> 
> 
> On Wed, 23 Dec 2020 at 09:10, Thomas Koenig <tkoenig@netcologne.de 
> <mailto:tkoenig@netcologne.de>> wrote:
> 
>     Hi Paul,
> 
>      > FAIL: gfortran.dg/caf-shared/scalar_alloc_1.f90 -pthread
>      > -fcoarray=shared  -O2  -lcaf_shared -lrt execution test
>      > FAIL: gfortran.dg/caf-shared/scalar_alloc_2.f90 -pthread  >
>     -fcoarray=shared  -O2  -lcaf_shared -lrt execution test
> 
>     I hope to have these patched today (depending on how the
>     Christmas preparations go :-)
> 
>      > FAIL: gfortran.dg/assumed_type_9.f90   -O2  execution test
>      > FAIL: gfortran.dg/assumed_type_9.f90   -Os  execution test
> 
>     That is an old failure, due to the fact that the branch hasn't
>     been updated to more recent trunk, where this no longer occurs.
> 
>     Best regards
> 
>              Thomas
> 
> 
> 
> -- 
> "If you can't explain it simply, you don't understand it well enough" - 
> Albert Einstein

Re: Update on shared memory coarrays

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

Hi everybody,

after Nicolas' offlineOK, I have now committed the attached patch
to the branch.

This should fix the test cases which are already committed
(hopefully...).  And I did add a FIXME for something that should
be cleaned up at a later date.

Regarding the failures that Steve reported, we'll try to reproduce
this using -m32 bits later - right now, Christmas is approching fast :-)

Best regards

	Thomas


Add offset to allocatable shared coarrays.

This adds the calculation of the offset for allocatable coarrays,
which was missing before, and fixes the resulting fallout for
ALLOCATED.  Additionally, it prepares the way for STAT and ERRMSG
for ALLOCATE of coarrays, but that still needs changes to
gfc_trans_allocate.

gcc/fortran/ChangeLog:

	* trans-array.c (gfc_conv_array_ref): If se->address_only is set,
	throw away all the offset calculation.
	(gfc_allocate_shared_coarray): Add arguments stat, errmsg and
	errlen to call to allocate.  Calculate offset for allocatable
	coarrays.
	(gfc_array_allocate): Adjust call to gfc_allocate_shared_coarray.
	* trans-array.h (gfc_allocate_shared_coarray): Change prototype
	of cas_coarray_alloc.
	* trans-decl.c (gfc_build_builtin_function_decls): Adjust
	cas_coarray_alloc to changed prototypes.
	(gfc_trans_shared_coarray): Adjust call to gfc_allocate_shared_coarray.
	* trans-intrinsic.c (gfc_conv_allocated): Set address_only on se.
	* trans.h: Add flag address_only to gfc_se.

libgfortran/ChangeLog:

	* caf_shared/wrapper.c (cas_coarray_alloc): Add status, error and
	errmsg arguments and their checking.

[-- Attachment #2: p5.diff --]
[-- Type: text/x-patch, Size: 8345 bytes --]

diff --git a/gcc/fortran/trans-array.c b/gcc/fortran/trans-array.c
index 39e6b6d9051..35afff5845e 100644
--- a/gcc/fortran/trans-array.c
+++ b/gcc/fortran/trans-array.c
@@ -3865,8 +3865,7 @@ gfc_conv_array_ref (gfc_se * se, gfc_array_ref * ar, gfc_expr *expr,
       add_to_offset (&cst_offset, &offset, tmp);
     }
 
-  if (flag_coarray == GFC_FCOARRAY_SHARED && need_impl_this_image
-      && !se->no_impl_this_image)
+  if (flag_coarray == GFC_FCOARRAY_SHARED && need_impl_this_image)
     {
       tree off;
       tree co_stride = gfc_conv_array_stride (decl, eff_dimen + 1);
@@ -3934,6 +3933,15 @@ gfc_conv_array_ref (gfc_se * se, gfc_array_ref * ar, gfc_expr *expr,
 	decl = NULL_TREE;
     }
 
+  /* Early return - only taken for ALLOCATED for shared coarrays.
+     FIXME - this could probably be done more elegantly.  */
+  if (se->address_only)
+    {
+      se->expr = build_array_ref (se->expr, build_int_cst (TREE_TYPE (offset), 0),
+				  decl, se->class_vptr);
+      return;
+    }
+
   se->expr = build_array_ref (se->expr, offset, decl, se->class_vptr);
 }
 
@@ -5975,15 +5983,41 @@ gfc_cas_get_allocation_type (gfc_symbol * sym)
 }
 
 void
-gfc_allocate_shared_coarray (stmtblock_t *b, tree decl, tree size, int corank,
-			    int alloc_type)
+gfc_allocate_shared_coarray (stmtblock_t *b, tree decl, tree size, int rank,
+			     int corank, int alloc_type, tree status,
+			     tree errmsg, tree errlen, bool calc_offset)
 {
+  tree st, err, elen;
+
+  if (status == NULL_TREE)
+    st = null_pointer_node;
+  else
+    st = gfc_build_addr_expr (NULL, status);
+
+  err = errmsg == NULL_TREE ? null_pointer_node : errmsg;
+  elen = errlen == NULL_TREE ? build_int_cst (gfc_charlen_type_node, 0) : errlen;
   gfc_add_expr_to_block (b,
 	build_call_expr_loc (input_location, gfor_fndecl_cas_coarray_allocate,
-			    4, gfc_build_addr_expr (pvoid_type_node, decl),
-			    size, build_int_cst (integer_type_node, corank),
-			    build_int_cst (integer_type_node, alloc_type)));
-
+			     7, gfc_build_addr_expr (pvoid_type_node, decl),
+			     size, build_int_cst (integer_type_node, corank),
+			     build_int_cst (integer_type_node, alloc_type),
+			     st, err, elen));
+  if (calc_offset)
+    {
+      int i;
+      tree offset, stride, lbound, mult;
+      offset = build_int_cst (gfc_array_index_type, 0);
+      for (i = 0; i < rank + corank; i++)
+	{
+	  stride = gfc_conv_array_stride (decl, i);
+	  lbound = gfc_conv_array_lbound (decl, i);
+	  mult = fold_build2_loc (input_location, MULT_EXPR,
+				  gfc_array_index_type, stride, lbound);
+	  offset = fold_build2_loc (input_location, MINUS_EXPR,
+				    gfc_array_index_type, offset, mult);
+	}
+      gfc_conv_descriptor_offset_set (b, decl, offset);
+    }
 }
 
 /* Initializes the descriptor and generates a call to _gfor_allocate.  Does
@@ -6193,7 +6227,9 @@ gfc_array_allocate (gfc_se * se, gfc_expr * expr, tree status, tree errmsg,
       int alloc_type
 	     = gfc_cas_get_allocation_type (expr->symtree->n.sym);
       gfc_allocate_shared_coarray (&elseblock, se->expr, elem_size,
-				   ref->u.ar.as->corank, alloc_type);
+				   ref->u.ar.as->rank, ref->u.ar.as->corank,
+				   alloc_type, status, errmsg, errlen,
+				   true);
     }
   /* The allocatable variant takes the old pointer as first argument.  */
   else if (allocatable)
diff --git a/gcc/fortran/trans-array.h b/gcc/fortran/trans-array.h
index 66f59bb068b..2168e9dc901 100644
--- a/gcc/fortran/trans-array.h
+++ b/gcc/fortran/trans-array.h
@@ -31,7 +31,8 @@ enum gfc_coarray_allocation_type {
 
 int gfc_cas_get_allocation_type (gfc_symbol *);
 
-void gfc_allocate_shared_coarray (stmtblock_t *, tree, tree, int, int);
+void gfc_allocate_shared_coarray (stmtblock_t *, tree, tree, int, int, int,
+				  tree, tree, tree, bool);
 
 /* Allow the bounds of a loop to be set from a callee's array spec.  */
 void gfc_set_loop_bounds_from_array_spec (gfc_interface_mapping *,
diff --git a/gcc/fortran/trans-decl.c b/gcc/fortran/trans-decl.c
index 91a5dca0c7f..f3526db7ea6 100644
--- a/gcc/fortran/trans-decl.c
+++ b/gcc/fortran/trans-decl.c
@@ -4118,9 +4118,15 @@ gfc_build_builtin_function_decls (void)
 	 get_identifier (PREFIX("cas_master")), ". r ", integer_type_node, 1,
 	build_pointer_type (build_function_type_list (void_type_node, NULL_TREE)));
       gfor_fndecl_cas_coarray_allocate = gfc_build_library_function_decl_with_spec (
-	 get_identifier (PREFIX("cas_coarray_alloc")), ". . R R R ", integer_type_node, 4,
-	pvoid_type_node, integer_type_node, integer_type_node, integer_type_node,
-	NULL_TREE);
+	 get_identifier (PREFIX("cas_coarray_alloc")), ". . R R R W W . ", integer_type_node, 7,
+	 pvoid_type_node,	/* desc.  */
+	 size_type_node,	/* elem_size.  */
+	 integer_type_node,	/* corank.  */
+	 integer_type_node,	/* alloc_type.  */
+	 pvoid_type_node,	/* stat.  */
+	 pvoid_type_node,	/* errmsg.  */
+	 gfc_charlen_type_node, /* errmsg_len.  */
+	 NULL_TREE);
       gfor_fndecl_cas_coarray_free = gfc_build_library_function_decl_with_spec (
 	 get_identifier (PREFIX("cas_coarray_free")), ". . R ", integer_type_node, 2,
 	 pvoid_type_node, /* Pointer to the descriptor to be deallocated.  */
@@ -4689,10 +4695,13 @@ gfc_trans_shared_coarray (stmtblock_t * init, stmtblock_t *cleanup, gfc_symbol *
 			   init, &overflow,
 			   NULL_TREE, &nelems, NULL,
 			   NULL_TREE, true, NULL, &element_size);
-      gfc_conv_descriptor_offset_set (init, decl, offset);
       elem_size = size_in_bytes (gfc_get_element_type (TREE_TYPE(decl)));
-      gfc_allocate_shared_coarray (init, decl, elem_size, sym->as->corank,
-				  alloc_type);
+      gfc_allocate_shared_coarray (init, decl, elem_size, sym->as->rank,
+				   sym->as->corank, alloc_type, null_pointer_node,
+				   null_pointer_node,
+				   build_int_cst (gfc_charlen_type_node, 0),
+				   false);
+      gfc_conv_descriptor_offset_set (init, decl, offset);
     }
 
   if (cleanup)
diff --git a/gcc/fortran/trans-intrinsic.c b/gcc/fortran/trans-intrinsic.c
index e93cd3a12c7..912c9b03a74 100644
--- a/gcc/fortran/trans-intrinsic.c
+++ b/gcc/fortran/trans-intrinsic.c
@@ -8832,7 +8832,7 @@ gfc_conv_allocated (gfc_se *se, gfc_expr *expr)
 	{
 	  /* Allocatable scalar.  */
 	  arg1se.want_pointer = 1;
-	  arg1se.no_impl_this_image = 1;
+	  arg1se.address_only = 1;
 	  gfc_conv_expr (&arg1se, arg1->expr);
 	  tmp = arg1se.expr;
 	}
diff --git a/gcc/fortran/trans.h b/gcc/fortran/trans.h
index f3cf33b342f..d3340b302ad 100644
--- a/gcc/fortran/trans.h
+++ b/gcc/fortran/trans.h
@@ -98,10 +98,9 @@ typedef struct gfc_se
      arrays in gfc_conv_expr_descriptor.  */
   unsigned use_offset:1;
 
-  /* For shared coarrays, do not add the offset for the implied
-     this_image().  */
-
-  unsigned no_impl_this_image:1;
+  /* Set if an array reference should be converted to an address of
+     its data pointer only.  */
+  unsigned address_only:1;
 
   unsigned want_coarray:1;
 
diff --git a/libgfortran/caf_shared/wrapper.c b/libgfortran/caf_shared/wrapper.c
index 471619441c0..a3d88660f01 100644
--- a/libgfortran/caf_shared/wrapper.c
+++ b/libgfortran/caf_shared/wrapper.c
@@ -44,7 +44,8 @@ enum gfc_coarray_allocation_type
   GFC_NCA_EVENT_COARRAY,
 };
 
-void cas_coarray_alloc (gfc_array_void *, int, int, int);
+void cas_coarray_alloc (gfc_array_void *, size_t, int, int, int *,
+			char *, size_t);
 export_proto (cas_coarray_alloc);
 
 void cas_coarray_free (gfc_array_void *, int);
@@ -85,8 +86,8 @@ void cas_collsub_broadcast_scalar (void *restrict, size_t, int, int *, char *,
 export_proto (cas_collsub_broadcast_scalar);
 
 void
-cas_coarray_alloc (gfc_array_void *desc, int elem_size, int corank,
-		   int alloc_type)
+cas_coarray_alloc (gfc_array_void *desc, size_t elem_size, int corank,
+		   int alloc_type, int *status, char *errmsg, size_t errmsg_len)
 {
   int i, last_rank_index;
   int num_coarray_elems, num_elems; /* Excludes the last dimension, because it
@@ -98,6 +99,7 @@ cas_coarray_alloc (gfc_array_void *desc, int elem_size, int corank,
   ensure_initialization (); /* This function might be the first one to be
 			       called, if it is called in a constructor.  */
 
+  STAT_ERRMSG_ENTRY_CHECK (status, errmsg, errmsg_len);
   if (alloc_type == GFC_NCA_LOCK_COARRAY)
     elem_size = sizeof (pthread_mutex_t);
   else if (alloc_type == GFC_NCA_EVENT_COARRAY)

Re: Update on shared memory coarrays

[Paul Richard Thomas]

Hi Thomas,

OK on the failures - I'll look out for the patch.

Is there a list somewhere of what has not been implemented but is
implemented in the existing coarrays library? Also, what is not implemented
from the standard?

Many thanks for your stupendous effort on this.

I am going to tackle the fixes for my two recent patches that have hit the
buffers PR83118 (patch as of last night) and PR98022 (patch that fixes it
but removes all the work that I have put into it :-( ). I'll post patches
today.

Paul


On Wed, 23 Dec 2020 at 09:10, Thomas Koenig <tkoenig@netcologne.de> wrote:

> Hi Paul,
>
> > FAIL: gfortran.dg/caf-shared/scalar_alloc_1.f90 -pthread
> > -fcoarray=shared  -O2  -lcaf_shared -lrt execution test
> > FAIL: gfortran.dg/caf-shared/scalar_alloc_2.f90 -pthread  >
> -fcoarray=shared  -O2  -lcaf_shared -lrt execution test
>
> I hope to have these patched today (depending on how the
> Christmas preparations go :-)
>
> > FAIL: gfortran.dg/assumed_type_9.f90   -O2  execution test
> > FAIL: gfortran.dg/assumed_type_9.f90   -Os  execution test
>
> That is an old failure, due to the fact that the branch hasn't
> been updated to more recent trunk, where this no longer occurs.
>
> Best regards
>
>         Thomas
>


-- 
"If you can't explain it simply, you don't understand it well enough" -
Albert Einstein

Re: Update on shared memory coarrays

[Thomas Koenig]

Hi Paul,

> FAIL: gfortran.dg/caf-shared/scalar_alloc_1.f90 -pthread 
> -fcoarray=shared  -O2  -lcaf_shared -lrt execution test
> FAIL: gfortran.dg/caf-shared/scalar_alloc_2.f90 -pthread  > -fcoarray=shared  -O2  -lcaf_shared -lrt execution test

I hope to have these patched today (depending on how the
Christmas preparations go :-)

> FAIL: gfortran.dg/assumed_type_9.f90   -O2  execution test
> FAIL: gfortran.dg/assumed_type_9.f90   -Os  execution test

That is an old failure, due to the fact that the branch hasn't
been updated to more recent trunk, where this no longer occurs.

Best regards

	Thomas

Re: Update on shared memory coarrays

[Paul Richard Thomas]

Hi Thomas and Nicolas,

I fear that, having been one of those that encouraged you into the shared
coarrays adventure, I just have not had the time to give it a whirl.
However, I just tried out the svn recipe that you cooked up for Steve. It
worked a treat but you should note the following failures on FC33/x86_64

FAIL: gfortran.dg/caf-shared/scalar_alloc_1.f90 -pthread -fcoarray=shared
 -O2  -lcaf_shared -lrt execution test
FAIL: gfortran.dg/caf-shared/scalar_alloc_2.f90 -pthread -fcoarray=shared
 -O2  -lcaf_shared -lrt execution test
FAIL: gfortran.dg/assumed_type_9.f90   -O2  execution test
FAIL: gfortran.dg/assumed_type_9.f90   -Os  execution test

Do you want some debugging done or are these failures old news?

I'll take this opportunity to wish everybody a happy holiday and a better
new year!

Paul




On Tue, 22 Dec 2020 at 15:11, Thomas Koenig via Fortran <fortran@gcc.gnu.org>
wrote:

>
> Hi Dominique,
>
> > Indeed! I can try to do some debugging under supervision.
> > Could you please tell me where to start?
>
> Thanks for your offer.
>
> The first thing to do would be to see where it hangs.  Generating
> a syscall trace (by whatever tool is used on MacOS, dtrace? ktrace?)
> would be a first step.
>
> A next step would be to compile a program with debug info and
> run it under a debugger, then see where it hangs.
>
> I'm not sure - do you use gdb on your system?  If you do,
> the following lines in your .gdbinit
>
> define better_mp_debug
>    set detach-on-fork off
>    set schedule-multiple on
>    set follow-fork-mode parent
>    set non-stop on
>    set target-async on
>    set print symbol-loading off
> end
>
> will give you a macro "better_mp_debug" which will ease debugging
> of forked processes enormously.
>
> My lack of experience with MacOS extends to which tools to use
> for debugging, unfortunately.
>
> Best regards
>
>         Thomas
>


-- 
"If you can't explain it simply, you don't understand it well enough" -
Albert Einstein

Re: Update on shared memory coarrays

[Thomas Koenig]

Hi Dominique,

> Indeed! I can try to do some debugging under supervision.
> Could you please tell me where to start?

Thanks for your offer.

The first thing to do would be to see where it hangs.  Generating
a syscall trace (by whatever tool is used on MacOS, dtrace? ktrace?)
would be a first step.

A next step would be to compile a program with debug info and
run it under a debugger, then see where it hangs.

I'm not sure - do you use gdb on your system?  If you do,
the following lines in your .gdbinit

define better_mp_debug
   set detach-on-fork off
   set schedule-multiple on
   set follow-fork-mode parent
   set non-stop on
   set target-async on
   set print symbol-loading off
end

will give you a macro "better_mp_debug" which will ease debugging
of forked processes enormously.

My lack of experience with MacOS extends to which tools to use
for debugging, unfortunately.

Best regards

	Thomas

Re: Update on shared memory coarrays

[dhumieres.dominique]

Le 2020-12-21 14:04, Thomas Koenig a écrit :
> Hi Dominique,
> 
> Like I said, hard to debug such a thing without access to the system 
> :-(
> 
> Best regards
> 
> 	Thomas

Indeed! I can try to do some debugging under supervision.
Could you please tell me where to start?

Cheers

Dominique

Re: Update on shared memory coarrays

[Thomas Koenig]

Hi Dominique,

> program hello_world
> implicit none
>    write(*,*) 'Hello world from ', &
>     this_image() , 'of', num_images()
> end program hello_world
> 
> does end, apparently looping in _gfortrani_shared_memory_init .

Is it possible to debug this further?

It works fine on Linux:

$ cat > hello.f90
program hello_world
implicit none
   write(*,*) 'Hello world from ', &
    this_image() , 'of', num_images()
end program hello_world
$ make hello
gfortran -S -O0 -Wall -fdump-tree-optimized -fdump-tree-original 
-fcoarray=shared -pthread -ggdb3 -fverbose-asm hello.f90
gfortran -c -O0 -Wall -fdump-tree-optimized -fdump-tree-original 
-fcoarray=shared -pthread -ggdb3 -fverbose-asm hello.s
gfortran -O0 -Wall -fdump-tree-optimized -fdump-tree-original 
-fcoarray=shared -pthread -ggdb3 -fverbose-asm hello.o -o hello 
/home/ig25/lib64/gcc/x86_64-pc-linux-gnu/11.0.0/libcaf_shared.a -lrt
$ ./hello
  Hello world from           14 of          16
  Hello world from           10 of          16
  Hello world from           16 of          16
  Hello world from            7 of          16
  Hello world from            9 of          16
  Hello world from            2 of          16
  Hello world from            5 of          16
  Hello world from            4 of          16
  Hello world from           13 of          16
  Hello world from           15 of          16
  Hello world from            1 of          16
  Hello world from            8 of          16
  Hello world from            3 of          16
  Hello world from            6 of          16
  Hello world from           12 of          16
  Hello world from           11 of          16

Like I said, hard to debug such a thing without access to the system :-(

Best regards

	Thomas

Re: Update on shared memory coarrays

[dhumieres.dominique]

Le 2020-12-03 08:27, Thomas Koenig a écrit :
> Am 02.12.20 um 18:41 schrieb dhumieres.dominique@free.fr:
>> Le 2020-11-30 23:13, Thomas Koenig a écrit :
>>> Hi Dominique,
>>> 
>>>> Is it enough?
>>> 
>>> Yep, it looks like it is working (to that extent).
>>> 
>>> Now, the next big question, how to write something for the
>>> testsuite that will execute test cases only when the
>>> library has been found, and will also link against that library
>>> (and against -lrt, if necessary...)
>>> 
>>> Best regards
>>> 
>>>     Thomas
>> 
>> Something such as check_effective_target_shared?
> 
> That is what I meant.
> 
>> Is there anything recursive in the caf_shared code?
> 
> As far as I remember, the library code is functions calling each
> other and loops. I don't think there is a recursive algorithm in
> there. Specifically, the reduction code for the collective
> subroutines is not recursive.
> 
> Best regards
> 
> 	Thomas

I asked the question, because running even a simple test:

program hello_world
implicit none
   write(*,*) 'Hello world from ', &
    this_image() , 'of', num_images()
end program hello_world

does end, apparently looping in _gfortrani_shared_memory_init .

TIA

Dominique

Re: Update on shared memory coarrays

[Thomas Koenig]

Am 02.12.20 um 18:41 schrieb dhumieres.dominique@free.fr:
> Le 2020-11-30 23:13, Thomas Koenig a écrit :
>> Hi Dominique,
>>
>>> Is it enough?
>>
>> Yep, it looks like it is working (to that extent).
>>
>> Now, the next big question, how to write something for the
>> testsuite that will execute test cases only when the
>> library has been found, and will also link against that library
>> (and against -lrt, if necessary...)
>>
>> Best regards
>>
>>     Thomas
> 
> Something such as check_effective_target_shared?

That is what I meant.

> Is there anything recursive in the caf_shared code?

As far as I remember, the library code is functions calling each
other and loops. I don't think there is a recursive algorithm in
there. Specifically, the reduction code for the collective
subroutines is not recursive.

Best regards

	Thomas

Re: Update on shared memory coarrays

[dhumieres.dominique]

Le 2020-11-30 23:13, Thomas Koenig a écrit :
> Hi Dominique,
> 
>> Is it enough?
> 
> Yep, it looks like it is working (to that extent).
> 
> Now, the next big question, how to write something for the
> testsuite that will execute test cases only when the
> library has been found, and will also link against that library
> (and against -lrt, if necessary...)
> 
> Best regards
> 
> 	Thomas

Something such as check_effective_target_shared?

Is there anything recursive in the caf_shared code?

Cheers,

Dominique

Re: Update on shared memory coarrays

[Thomas Koenig]

Hi Dominique,

> Is it enough?

Yep, it looks like it is working (to that extent).

Now, the next big question, how to write something for the
testsuite that will execute test cases only when the
library has been found, and will also link against that library
(and against -lrt, if necessary...)

Best regards

	Thomas

Re: Update on shared memory coarrays

[dhumieres.dominique]

Le 2020-11-30 07:21, Thomas Koenig a écrit :
> Hi Dominique,
> 
> thanks for your testing.
> 
>> AFAIU no test for -lcaf_shared is actually run in the test suite, so I 
>> tried to do some testing manually
>> and my tests failed: zillion calls to _gfortrani_shared_memory_init.
> 
> So, apparently the library was not found.  Is it built?

Indeed!

> 
> Can you build and execute a "Hello world" program with -fcoarray=shared
> if you supply the right flags to link against the library and
> against librt (or whatever is needed for shm_open, if anything)?
> 
> Best regards
> 
> 	Thomas

% /opt/gcc/gcc11co/bin/gfortran 
/opt/gcc/coarray/gcc/testsuite/gfortran.dg/coarray_41.f90 -fcoarray=lib 
-lcaf_single
% time ./a.out
0.000u 0.000s 0:00.12 0.0%	0+0k 0+0io 0pf+0w
% /opt/gcc/gcc11co/bin/gfortran 
/opt/gcc/coarray/gcc/testsuite/gfortran.dg/coarray_41.f90 
-fcoarray=shared -lcaf_shared
% time ./a.out
^C30.586u 12.860s 0:43.73 99.3%	0+0k 0+0io 0pf+0w

Is it enough?

Dominique

Re: Update on shared memory coarrays

[Thomas Koenig]

Hi Dominique,

thanks for your testing.

> AFAIU no test for -lcaf_shared is actually run in the test suite, so I 
> tried to do some testing manually
> and my tests failed: zillion calls to _gfortrani_shared_memory_init.

So, apparently the library was not found.  Is it built?

Can you build and execute a "Hello world" program with -fcoarray=shared
if you supply the right flags to link against the library and
against librt (or whatever is needed for shm_open, if anything)?

Best regards

	Thomas

Update on shared memory coarrays

[dhumieres.dominique]

Hi Nocolas,

> * Use of pthread_barrier_t and associated functions has been removed,
> so there is a chance that the library compiles on MacOS now.

I confirmed that the new version bootstrap on MacOS (Big Sur).

I have run make check-fortran and got the following failures:

FAIL: gfortran.dg/assumed_type_9.f90   -O2  execution test
FAIL: gfortran.dg/assumed_type_9.f90   -Os  execution test
FAIL: gfortran.dg/backtrace_1.f90   -O0  execution test
FAIL: gfortran.dg/backtrace_1.f90   -O1  execution test
FAIL: gfortran.dg/backtrace_1.f90   -O2  execution test
FAIL: gfortran.dg/backtrace_1.f90   -O3 -fomit-frame-pointer 
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/backtrace_1.f90   -O3 -g  execution test
FAIL: gfortran.dg/backtrace_1.f90   -Os  execution test
FAIL: libgomp.fortran/examples-4/simd-2.f90   -O1  (internal compiler 
error)
FAIL: libgomp.fortran/examples-4/simd-2.f90   -O1  (test for excess 
errors)
FAIL: libgomp.fortran/examples-4/simd-2.f90   -Os  (internal compiler 
error)
FAIL: libgomp.fortran/examples-4/simd-2.f90   -Os  (test for excess 
errors)
FAIL: libgomp.fortran/examples-4/simd-3.f90   -O1  execution test
FAIL: gfortran.dg/widechar_intrinsics_9.f90   -O0  execution test
FAIL: gfortran.dg/widechar_intrinsics_9.f90   -O1  execution test
FAIL: gfortran.dg/widechar_intrinsics_9.f90   -O2  execution test
FAIL: gfortran.dg/widechar_intrinsics_9.f90   -O3 -fomit-frame-pointer 
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/widechar_intrinsics_9.f90   -O3 -g  execution test
FAIL: gfortran.dg/widechar_intrinsics_9.f90   -Os  execution test

The tests gfortran.dg/assumed_type_9.f90 and 
libgomp.fortran/examples-4/simd-* are now fixed.
The failures for gfortran.dg/backtrace_1.f90 and 
widechar_intrinsics_9.f90 are probably due to Big Sur (pr97865).

AFAIU no test for -lcaf_shared is actually run in the test suite, so I 
tried to do some testing manually
and my tests failed: zillion calls to _gfortrani_shared_memory_init.

For testing one could try to add

     foreach flags $option_list {
	verbose "Testing $nshort (libcaf_shared), $flags" 1
         set gfortran_aux_module_flags "-fcoarray=lib $flags 
-lcaf_shared"
	dg-test $test "-fcoarray=lib $flags -lcaf_shared $maybe_atomic_lib" ""
	cleanup-modules ""
     }

near the end of gfortran.dg/coarray/caf.exp.

It would also nice to have all the carry tests in gfortran.dg moved to 
the coarray subdirectory.

One could also use the tests in opencoarray.

Thanks for the work,

Dominique