From: Tobias Burnus <tobias.burnus@physik.fu-berlin.de>
To: Matthew Wahab <matthew.wahab@foss.arm.com>
Cc: fortran@gcc.gnu.org, gcc-patches@gcc.gnu.org,
Alessandro Fanfarillo <fanfarillo.gcc@gmail.com>
Subject: Re: [Fortran, Patch] Memory sync after coarray image control statements and assignment
Date: Tue, 08 Dec 2015 09:26:00 -0000 [thread overview]
Message-ID: <20151208092556.GB29964@physik.fu-berlin.de> (raw)
In-Reply-To: <56659312.6090700@foss.arm.com>
On Mon, Dec 07, 2015 at 02:09:22PM +0000, Matthew Wahab wrote:
> >>I wonder whether using
> >>__asm__ __volatile__ ("":::"memory");
> >>would be sufficient as it has a way lower overhead than
> >>__sync_synchronize().
>
> I don't know anything about Fortran or coarrays and I'm curious
> whether this affects architectures with weak memory models. Is the
> barrier only needed to stop reordering by the compiler or is does it
> also need to stop reordering by the hardware?
Short answer: I think no mfence is needed as either the communication
is local (to the thread/process) - in which case the hardware will act
correctly - or the communication is remote (different thread, process,
communication to different computer via interlink [ethernet, infiniband,
...]); and in the later case, the communication library has to deal with
it.
However, I think that except for SYNC using
__asm__ __volatile__ ("":::"memory");
is the wrong solution - even though it might work as band aid.
* * *
To gether some suggestions, here is how coarrays work. They are based on:
everything is a local variable - except it is a coarray. And all communication
is explicit - but is often single sideded. That scheme works well (also)
with distributed memory and is similar to MPI (Message Passing Interface).
Using
integer :: var[*]
one declares a coarray scalar variable. Accessing it as
var = 5
tmp = var
acts on the variable on the current 'image' (process). Using
var[idx] = 5 ! Set 'var' in image 'idx' to '5'
tmp = var[idx] ! Obtain value of 'var' on image 'idx'
one accesses that variable on a remote image - except if 'idx' matches
the current image; in that case, it acts on the local variable.
In one segment (which ends at explicit or implicit synchronizations,
e.g. using SYNC ALL): If the value of a is changed - either locally
or by a remote image - then only that image is permitted to 'read' the
value (or to change it as that would be another possibility for race
conditions).
Very often, coarray variables are arrays and heavily accessed as local
varable in hot loops. But on the other hand, the value of the variable
can be changed by external means - up to having hardware support to
write into the memory from another computer.
Thus, there two cases were the local view might fail:
(a) when the variable has been changed in a previous segment by a
remote process, e.g.
var = 5 ! assue 'var' is a coarray
sync all ! end of segment
! ! var is changed by a remote image
sync all ! end of segment
tmp = var
where "var" might or might not have the value 5. Or more fancy
without locks and global barriers:
type(event_type) :: var_is_set[*]
var = 5
event post(have_set_var[remote_idx])
! Remote waits for event, sets out 'var' to 42 and
! then posts an event that it is set
event wait(have_set_var)
tmp = var
where one tells the remote process that "var = 5" has been set
and waits until it has set the local variable "var".
(b) when the variable is changed on the same image by two means, e.g.
var = 5
var[this_image()] = 42
tmp = var
The communication maps to function calls like:
__gfortran_caf_send(var_token, idx, 5) // var[idx] = 5
__gfortran_caf_get (tmp, var_token, idx) // tmp = var[idx]
[Real commands, see
https://gcc.gnu.org/onlinedocs/gfortran/index.html#toc_Coarray-Programming]
It is assumed that the library called takes care of the hardware part,
i.e. locking, mfence etc. - such that in the compiler itself, one only
has to deal with restricting compiler optimizations to the places where
it is permitted.
A coarray comes into existance via:
var = _gfortran_caf_register (size, &var_token);
similar to malloc - and var_token identifies the coarray; what the
content is, is left to the library. - For the good or worse,
_gfortran_caf_register doesn't tell the compiler that it has clobbered
"var" as in a way the pointer address escaped. Nor is "var" marked as
volatile.
Coming back to item (a): After a segment has ended (SYNC ALL, EVENT WAIT
or similar), the compiler can no longer assume that coarray variables
have the same value. Those variables can be hidden as in
call foo()
sync all
call foo()
where 'foo' doesn't know about the 'sync all' and the caller doesn't know
whether 'foo' has accesses to coarrays (and whether it accesses them). I
think
__asm__ __volatile__ ("":::"memory");
should be sufficient in that case, assuming that the communication library
takes are of making all changes available (mfence, __sync_synchronize or
whatever).
The other question is item (b): Here, one has an alias problem within the
segment - but the alias only rarely happens. Code of the form
var[idx] = ...
usually does not access the local memory; it happens only in corner cases
like:
do i = 1, num_images
var[i] = 5
end do
which sets 'var' to 5 on all images - including the local image. This will
produce code like (full example below):
*var = 5
_gfortran_caf_send(var_token, idx, 42);
tmp = *var
where the frontend knows that _gfortran_caf_send might modify 'var' iff idx
matches the local image. The question is only how to tell this the compiler.
Using __asm__ __volatile__ ("":::"memory"); seems to be appropriate, even
though something more tailored would be nice, something like:
__asm__ __volatile__ ("": : "+rm" (*var) : "")
but I am not sure whether that's valid and works - additionally, it assumes
that one reads "*var" (i.e. that it is defined before), which is not required.
(No idea whether -Wundefined would trigger.)
I there some better way to express this?
Cheers,
Tobias
PS: Full example:
integer :: tmp
integer :: var[*]
var = 5
var[this_image()] = 42
tmp = var
end
This will generate a static constructor, run at program startup:
_caf_init.1 ()
{ // v-- allocate 4 byte
var = _gfortran_caf_register (4, 0, &caf_token.0, 0B, 0B, 0);
}
and the (main) program code (slightly trimmed):
static void * restrict caf_token.0;
static integer(kind=4) * restrict var;
void _caf_init.1 (void);
*var = 4;
desc.3.data = 42;
_gfortran_caf_send (caf_token.0, 0B /* offset */ var,
_gfortran_caf_this_image (0), &desc.2, 0B, &desc.3, 4, 4, 0);
__asm__ __volatile__("":::"memory"); // new
tmp = *var;
The problem is that in that case the compiler does not know that
"_gfortran_caf_send (caf_token.0," can modify "*var".
next prev parent reply other threads:[~2015-12-08 9:26 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-06 21:02 Alessandro Fanfarillo
2015-12-07 7:20 ` Tobias Burnus
2015-12-07 9:16 ` Alessandro Fanfarillo
2015-12-07 10:06 ` Tobias Burnus
2015-12-07 14:09 ` Matthew Wahab
2015-12-08 9:26 ` Tobias Burnus [this message]
2015-12-09 14:47 ` Matthew Wahab
2015-12-07 14:48 ` Alessandro Fanfarillo
2015-12-08 10:01 ` Tobias Burnus
2015-12-08 14:21 ` Alessandro Fanfarillo
2015-12-09 7:23 ` Tobias Burnus
2015-12-09 16:08 ` Alessandro Fanfarillo
2015-12-09 22:16 ` Tobias Burnus
2015-12-10 8:44 ` Alessandro Fanfarillo
[not found] ` <20151210090345.GB6991@physik.fu-berlin.de>
[not found] ` <CAHqFgjVhx6pcVFEaCT8=9P+vkR=SP-mohZwr+B315N0_41OtKA@mail.gmail.com>
2015-12-14 19:14 ` Tobias Burnus
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151208092556.GB29964@physik.fu-berlin.de \
--to=tobias.burnus@physik.fu-berlin.de \
--cc=fanfarillo.gcc@gmail.com \
--cc=fortran@gcc.gnu.org \
--cc=gcc-patches@gcc.gnu.org \
--cc=matthew.wahab@foss.arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).