From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailg210.ethz.ch (mailg210.ethz.ch [IPv6:2001:67c:10ec:5606::21]) by sourceware.org (Postfix) with ESMTPS id 191493892447 for ; Tue, 5 Jan 2021 13:55:19 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 191493892447 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=student.ethz.ch Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=koenigni@student.ethz.ch Received: from mailm211.d.ethz.ch (2001:67c:10ec:5603::25) by mailg210.ethz.ch (2001:67c:10ec:5606::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Tue, 5 Jan 2021 14:54:57 +0100 Received: from [10.5.35.95] (10.5.35.95) by mailm211.d.ethz.ch (2001:67c:10ec:5603::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Tue, 5 Jan 2021 14:55:17 +0100 Subject: Re: Update on shared memory coarrays To: Iain Sandoe , Thomas Koenig CC: Fortran List , "dhumieres.dominique@free.fr" References: <0770e8fb5b4b479afc98be8cf6221cc5@free.fr> <1a63faebd65be2a76627a7873d74c0c8@free.fr> <86fa53e3-155d-1912-8b88-b0789323ee09@netcologne.de> <2d94ef13-e12e-b941-dcb0-a9a79362f2d5@netcologne.de> <6D26C802-E6E0-473F-BB17-A9B373541980@sandoe.co.uk> <00ABD0DC-235B-4A4E-B9E0-B1200AC07EAD@sandoe.co.uk> <191C9834-F907-4CF0-B652-16E04A42B7AB@sandoe.co.uk> <5683F724-2723-4EE4-AC50-83FDB598D238@sandoe.co.uk> <2007278F-63EE-427A-BFB6-A504E96D3572@sandoe.co.uk> <116ed7ff-7cdc-6324-c6c3-33fa9b1a036c@netcologne.de> <72972030-407F-4BD7-9B6C-D6937292E906@sandoe.co.uk> <816E309C-4983-4159-973D-F1A46AD8CB3C@sandoe.co.uk> From: =?UTF-8?Q?Nicolas_K=c3=b6nig?= Message-ID: <4020eab0-cd40-7ed5-5de0-ebb0727815f9@student.ethz.ch> Date: Tue, 5 Jan 2021 15:54:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <816E309C-4983-4159-973D-F1A46AD8CB3C@sandoe.co.uk> Content-Type: multipart/mixed; boundary="------------B36CBC586AAB2E25BC21ED06" Content-Language: en-US X-Originating-IP: [10.5.35.95] X-ClientProxiedBy: mailm217.d.ethz.ch (2001:67c:10ec:5603::31) To mailm211.d.ethz.ch (2001:67c:10ec:5603::25) X-TM-SNTS-SMTP: DA05F1D8012D4806D80E86AB89A25381A1A126203370DCFA78E70925C091B4F62000:8 X-Spam-Status: No, score=-0.6 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, LIKELY_SPAM_FROM, NICE_REPLY_A, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: fortran@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Fortran mailing list List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jan 2021 13:55:21 -0000 --------------B36CBC586AAB2E25BC21ED06 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit Hi Iain, On 05/01/2021 14:11, Iain Sandoe wrote: > Hi folks, > > Nicolas König wrote: >> On 04/01/2021 03:34, Iain Sandoe wrote: >>> Thomas Koenig via Fortran wrote: >>>> Am 03.01.21 um 16:42 schrieb Iain Sandoe: > >>> It seems shm is not a favoured IPC mechanism for Darwin (it’s possibly >>> disallowed for GUI apps, according to search results - I didn’t check >>> that) >>>>> (we can file a radar, and maybe it gets fixed at some point - but >>>>> that’s no help to all the existing system versions) >>>> >>>> Makes sense to file a bug report with Apple.  And yes, we'll have to >>>> deal with this some other way. >>> [for command line apps] Either mechanism for accessing shm works; >>> [1] shm_open / fd / mmap / munmap >>> [2] shmget / shmat / shmdt <= allocation is rounded up to the nearest >>> page boundary. >>> Both have the constraint [on Darwin] that the size of the allocation >>> cannot be increased after it is set (by ftruncate() [1] or shmget() [2]). >>> It can be decreased (by [2] only), but it’s not clear if that actually >>> has any physical effect. >> >> That's a problem, since coarrays can be allocated at runtime, and we >> can't possibly precompute their size. > > OK. > >> To get coarrays to work, we would probably need to create new shared >> memory objects (with predictable names, so they can be opened from other >> images), but that would be a significant time investment. > > How does it work between different images with shared memory unless the > name to shm_open () is predictable between images? Currently, the images are launched by one "master"-process, that also handles all the management. There, we can get some shared memory via mmap(MAP_ANON | MAP_SHARED), so the master could communicate the base name of the image. At the moment, it just opens a shared_memory object and passes the file descriptor to its children. > > (there has to be some shared key of some form, right?) > >> It also doesn't bring my hopes up that other things like >> pthread_{mutex,cond}attr_setpshared() will work. > > Darwin is Posix SUSv3 (actually, AFAICT the majority of the certified > platforms are; there only seems to be one certified to UNIX7). > > - so pthreads stuff that’s mandated in SUSv3 is definitely present on > Darwin. > > (in practice some of the optional stuff is also present - you can look at > what libstdc++ does to get an idea of what works in a cross-platform manner) Process shared mutexes/condition variables are optional :) Attached is a test case that should check whether the two work. > >> I think it's probably better to focus the attention on getting it to work >> on Linux/BSD. > > Well, I can understand making progress in that way - but it’s a bit of a > slippery slope towards an “x86 Linux only” compiler (if we keep taking that > route until a significant portion of the useful features don’t work > elsewhere). As a person that uses a ppc64le machine as his work station, I can completely understand that concern. > > Of course, this particular case is not so exclusive [it appears from > previous mails that Darwin is the only one with an issue], but as a general > rule it would be good to pick an impl that works on at least the regularly > used/tested targets. > >>> I guess that one ought to consider that a change in allocation could >>> trigger a copy on any system (since there might not be contiguous memory >>> pages available for an arbitrary increase) - so a strategy which either >>> allocates separae fragments for each need - or waits until the size is >>> known before allocation is likely to perform better on average, >> >> There's no need to have the continuous virtual memory backed by >> continuous physical memory, so nothing here should trigger a relocation >> of the actual physical memory. There are just some entries in the page >> table pointing to the same pages. > > I guess if everything is done on page boundaries & sizes, that can work - > providing one doesn’t mind the base pointer changing across resizes. > > ==== > > Update: > > three codes attached. > > 1. Posix shm implementation; works fine but is not resizeable. > > 2. Darwin shm implementation : works fine but is only resizable downwards > and it’s not clear if the resources are actually released. At the moment, the shared memory is never unmapped but instead handed to an allocator, so the ability to make it smaller is optional. Either of these should work, if we create more than one. They should also give better perf than 3. > > 3. mmapped shared file : This works [on Darwin at least] and is resizable. > I would assume that, in reality, all accesses would be to the buffer cache > [unless something actually forces the cache to be flushed] and therefore > that the performance ought to be similar to shm. This would probably be the fastest way to get some implementation to work, but the page fault/context switch at each access to a coarray hurts. > > Worst case, one could point /tmp at a ramdisk .. > (I tend to do that anyway, I don’t want my sshds getting hammered by > toolchain builds). > > Thoughts? I think the best way to proceed is to introduce a new level of abstraction over the shared memory object that makes it easier to swap an alternative implementation depending on the system (I think the current shared_memory.c/h still has too much other logic in it, but I'm not sure). Nicolas > Iain > > > > --------------B36CBC586AAB2E25BC21ED06 Content-Type: text/x-csrc; charset="UTF-8"; name="shared_cond_test.c" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="shared_cond_test.c" #include #include #include #include #include #include #include #define ERR_CHK(x) \ do { \ if (x) { \ perror(#x "failed"); \ exit(1); \ } \ } while (0) int main(int argc, char **argv) { pthread_condattr_t cattr; pthread_mutexattr_t mattr; pthread_mutex_t *m; pthread_cond_t *c; void *mem; ERR_CHK((mem = mmap(0, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANON, -1, 0)) == MAP_FAILED); m = mem; c = mem + 0x800; ERR_CHK(pthread_mutexattr_init(&mattr)); ERR_CHK(pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED)); ERR_CHK(pthread_mutex_init(m, &mattr)); ERR_CHK(pthread_mutexattr_destroy(&mattr)); ERR_CHK(pthread_condattr_init(&cattr)); ERR_CHK(pthread_condattr_setpshared(&cattr, PTHREAD_PROCESS_SHARED)); ERR_CHK(pthread_cond_init(c, &cattr)); ERR_CHK(pthread_condattr_destroy(&cattr)); printf("Success\n"); return 0; } --------------B36CBC586AAB2E25BC21ED06--