Fwd: [PATCH 08/25] Fix co-array allocation

Fwd: [PATCH 08/25] Fix co-array allocation

[Toon Moene]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]




-------- Forwarded Message --------
Subject: [PATCH 08/25] Fix co-array allocation
Date: Wed, 5 Sep 2018 12:49:40 +0100
From: ams@codesourcery.com
To: gcc-patches@gcc.gnu.org


The Fortran front-end has a bug in which it uses "int" values for "size_t"
parameters.  I don't know why this isn't problem for all 64-bit 
architectures,
but GCN ends up with the data in the wrong argument register and/or 
stack slot,
and bad things happen.

This patch corrects the issue by setting the correct type.

2018-09-05  Kwok Cheung Yeung  <kcy@codesourcery.com>

	gcc/fortran/
	* trans-expr.c (gfc_trans_structure_assign): Ensure that
	integer_zero_node is of sizetype when used as the first
	argument of a call to _gfortran_caf_register.
	* trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
	index to a size_t type.
	* trans-stmt.c (gfc_trans_event_post_wait): Likewise.
---
  gcc/fortran/trans-expr.c      | 2 +-
  gcc/fortran/trans-intrinsic.c | 3 ++-
  gcc/fortran/trans-stmt.c      | 3 ++-
  3 files changed, 5 insertions(+), 3 deletions(-)



[-- Attachment #2: 0008-Fix-co-array-allocation.patch --]
[-- Type: text/x-patch, Size: 1832 bytes --]

diff --git a/gcc/fortran/trans-expr.c b/gcc/fortran/trans-expr.c
index 56ce98c..91be3fb 100644
--- a/gcc/fortran/trans-expr.c
+++ b/gcc/fortran/trans-expr.c
@@ -7729,7 +7729,7 @@ gfc_trans_structure_assign (tree dest, gfc_expr * expr, bool init, bool coarray)
 		 suffices to recognize the data as array.  */
 	      if (rank < 0)
 		rank = 1;
-	      size = integer_zero_node;
+	      size = fold_convert (sizetype, integer_zero_node);
 	      desc = field;
 	      gfc_add_modify (&block, gfc_conv_descriptor_rank (desc),
 			      build_int_cst (signed_char_type_node, rank));
diff --git a/gcc/fortran/trans-intrinsic.c b/gcc/fortran/trans-intrinsic.c
index b2cea93..23c13da 100644
--- a/gcc/fortran/trans-intrinsic.c
+++ b/gcc/fortran/trans-intrinsic.c
@@ -10732,7 +10732,8 @@ conv_intrinsic_event_query (gfc_code *code)
 	      tmp = fold_build2_loc (input_location, MULT_EXPR,
 				     integer_type_node, extent, tmp);
 	      index = fold_build2_loc (input_location, PLUS_EXPR,
-				       integer_type_node, index, tmp);
+				       size_type_node, index,
+				       fold_convert (size_type_node, tmp));
 	      if (i < ar->dimen - 1)
 		{
 		  ubound = gfc_conv_descriptor_ubound_get (desc, gfc_rank_cst[i]);
diff --git a/gcc/fortran/trans-stmt.c b/gcc/fortran/trans-stmt.c
index 795d3cc..2c59675 100644
--- a/gcc/fortran/trans-stmt.c
+++ b/gcc/fortran/trans-stmt.c
@@ -1096,7 +1096,8 @@ gfc_trans_event_post_wait (gfc_code *code, gfc_exec_op op)
 	  tmp = fold_build2_loc (input_location, MULT_EXPR,
 				 integer_type_node, extent, tmp);
 	  index = fold_build2_loc (input_location, PLUS_EXPR,
-				   integer_type_node, index, tmp);
+				   size_type_node, index,
+				   fold_convert (size_type_node, tmp));
 	  if (i < ar->dimen - 1)
 	    {
 	      ubound = gfc_conv_descriptor_ubound_get (desc, gfc_rank_cst[i]);

Re: [PATCH 08/25] Fix co-array allocation

[Bernhard Reutner-Fischer]

On Wed, 5 Sep 2018 at 18:54, Toon Moene <toon@moene.org> wrote:
>
>

mhm, but why don't these use size_zero_node ?

thanks,
>
>
> -------- Forwarded Message --------
> Subject: [PATCH 08/25] Fix co-array allocation
> Date: Wed, 5 Sep 2018 12:49:40 +0100
> From: ams@codesourcery.com
> To: gcc-patches@gcc.gnu.org
>
>
> The Fortran front-end has a bug in which it uses "int" values for "size_t"
> parameters.  I don't know why this isn't problem for all 64-bit
> architectures,
> but GCN ends up with the data in the wrong argument register and/or
> stack slot,
> and bad things happen.
>
> This patch corrects the issue by setting the correct type.
>
> 2018-09-05  Kwok Cheung Yeung  <kcy@codesourcery.com>
>
>         gcc/fortran/
>         * trans-expr.c (gfc_trans_structure_assign): Ensure that
>         integer_zero_node is of sizetype when used as the first
>         argument of a call to _gfortran_caf_register.
>         * trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
>         index to a size_t type.
>         * trans-stmt.c (gfc_trans_event_post_wait): Likewise.
> ---
>   gcc/fortran/trans-expr.c      | 2 +-
>   gcc/fortran/trans-intrinsic.c | 3 ++-
>   gcc/fortran/trans-stmt.c      | 3 ++-
>   3 files changed, 5 insertions(+), 3 deletions(-)
>
>

Re: [PATCH 08/25] Fix co-array allocation

[Janne Blomqvist]

Please send fortran patches to the fortran list as well!

On Wed, Sep 5, 2018 at 7:54 PM Toon Moene <toon@moene.org> wrote:

>
>
>
> -------- Forwarded Message --------
> Subject: [PATCH 08/25] Fix co-array allocation
> Date: Wed, 5 Sep 2018 12:49:40 +0100
> From: ams@codesourcery.com
> To: gcc-patches@gcc.gnu.org
>
>
> The Fortran front-end has a bug in which it uses "int" values for "size_t"
> parameters.  I don't know why this isn't problem for all 64-bit
> architectures,
> but GCN ends up with the data in the wrong argument register and/or
> stack slot,
> and bad things happen.
>
> This patch corrects the issue by setting the correct type.
>
> 2018-09-05  Kwok Cheung Yeung  <kcy@codesourcery.com>
>
>         gcc/fortran/
>         * trans-expr.c (gfc_trans_structure_assign): Ensure that
>         integer_zero_node is of sizetype when used as the first
>         argument of a call to _gfortran_caf_register.
>

The argument must be of type size_type_node, not sizetype. Please instead
use

size = build_zero_cst (size_type_node);


>         * trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
>         index to a size_t type.
>

Using integer_type_node is wrong, but the correct type for calculating
array indices (lbound, ubound,  etc.) is not size_type_node but rather
gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
that, and then fold_convert index to size_type_node just before generating
the call to event_query.


>         * trans-stmt.c (gfc_trans_event_post_wait): Likewise.
>

Same here as above.

Thanks,
-- 
Janne Blomqvist

Re: [PATCH 08/25] Fix co-array allocation

[Andrew Stubbs]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

On 05/09/18 19:07, Janne Blomqvist wrote:
> The argument must be of type size_type_node, not sizetype. Please instead
> use
> 
> size = build_zero_cst (size_type_node);
> 
> 
>>          * trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
>>          index to a size_t type.
>>
> 
> Using integer_type_node is wrong, but the correct type for calculating
> array indices (lbound, ubound,  etc.) is not size_type_node but rather
> gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
> that, and then fold_convert index to size_type_node just before generating
> the call to event_query.
> 
> 
>>          * trans-stmt.c (gfc_trans_event_post_wait): Likewise.
>>
> 
> Same here as above.

How is the attached? I retested and found no regressions.

Andrew

[-- Attachment #2: 180919-fix-co-array-allocation.patch --]
[-- Type: text/x-patch, Size: 3341 bytes --]

Fix co-array allocation

The Fortran front-end has a bug in which it uses "int" values for "size_t"
parameters.  I don't know why this isn't problem for all 64-bit architectures,
but GCN ends up with the data in the wrong argument register and/or stack slot,
and bad things happen.

This patch corrects the issue by setting the correct type.

2018-09-19  Andrew Stubbs  <ams@codesourcery.com>
            Kwok Cheung Yeung  <kcy@codesourcery.com>

	gcc/fortran/
	* trans-expr.c (gfc_trans_structure_assign): Ensure that the first
	argument of a call to _gfortran_caf_register is of size_type_node.
	* trans-intrinsic.c (conv_intrinsic_event_query): Convert computed
	index to a size_type_node type.
	* trans-stmt.c (gfc_trans_event_post_wait): Likewise.

diff --git a/gcc/fortran/trans-expr.c b/gcc/fortran/trans-expr.c
index 56ce98c..28079ac 100644
--- a/gcc/fortran/trans-expr.c
+++ b/gcc/fortran/trans-expr.c
@@ -7729,7 +7729,7 @@ gfc_trans_structure_assign (tree dest, gfc_expr * expr, bool init, bool coarray)
 		 suffices to recognize the data as array.  */
 	      if (rank < 0)
 		rank = 1;
-	      size = integer_zero_node;
+	      size = build_zero_cst (size_type_node);
 	      desc = field;
 	      gfc_add_modify (&block, gfc_conv_descriptor_rank (desc),
 			      build_int_cst (signed_char_type_node, rank));
diff --git a/gcc/fortran/trans-intrinsic.c b/gcc/fortran/trans-intrinsic.c
index b2cea93..569435d 100644
--- a/gcc/fortran/trans-intrinsic.c
+++ b/gcc/fortran/trans-intrinsic.c
@@ -10732,7 +10732,9 @@ conv_intrinsic_event_query (gfc_code *code)
 	      tmp = fold_build2_loc (input_location, MULT_EXPR,
 				     integer_type_node, extent, tmp);
 	      index = fold_build2_loc (input_location, PLUS_EXPR,
-				       integer_type_node, index, tmp);
+				       gfc_array_index_type, index,
+				       fold_convert (gfc_array_index_type,
+						     tmp));
 	      if (i < ar->dimen - 1)
 		{
 		  ubound = gfc_conv_descriptor_ubound_get (desc, gfc_rank_cst[i]);
@@ -10756,6 +10758,7 @@ conv_intrinsic_event_query (gfc_code *code)
 	  stat = gfc_create_var (integer_type_node, "stat");
 	}
 
+      index = fold_convert (size_type_node, index);
       tmp = build_call_expr_loc (input_location, gfor_fndecl_caf_event_query, 5,
                                    token, index, image_index, count
 				   ? gfc_build_addr_expr (NULL, count) : count,
diff --git a/gcc/fortran/trans-stmt.c b/gcc/fortran/trans-stmt.c
index 795d3cc..92d9c37 100644
--- a/gcc/fortran/trans-stmt.c
+++ b/gcc/fortran/trans-stmt.c
@@ -1096,7 +1096,8 @@ gfc_trans_event_post_wait (gfc_code *code, gfc_exec_op op)
 	  tmp = fold_build2_loc (input_location, MULT_EXPR,
 				 integer_type_node, extent, tmp);
 	  index = fold_build2_loc (input_location, PLUS_EXPR,
-				   integer_type_node, index, tmp);
+				   gfc_array_index_type, index,
+				   fold_convert (gfc_array_index_type, tmp));
 	  if (i < ar->dimen - 1)
 	    {
 	      ubound = gfc_conv_descriptor_ubound_get (desc, gfc_rank_cst[i]);
@@ -1130,6 +1131,7 @@ gfc_trans_event_post_wait (gfc_code *code, gfc_exec_op op)
       stat = gfc_create_var (integer_type_node, "stat");
     }
 
+  index = fold_convert (size_type_node, index);
   if (op == EXEC_EVENT_POST)
     tmp = build_call_expr_loc (input_location, gfor_fndecl_caf_event_post, 6,
 			       token, index, image_index,

Re: [PATCH 08/25] Fix co-array allocation

[Damian Rouson]

Has this been tested in multi-image execution using OpenCoarrays?   If not,
I would be glad to assist with installing OpenCoarrays so that it can be
part of the testing process.

On a related note, two Sourcery Institute developers have attempted to edit
the GCC build system to make the downloading and building of OpenCoarrays
automatically part of the gfortran build process.  Neither developer
succeeded.  If anyone has any interest in figuring out how to do this, it
will prevent a lot of potential regressions when single-image testing
doesn't expose issues that only arise with multi-image execution.

Damian

On Wed, Sep 19, 2018 at 9:25 AM Andrew Stubbs <ams@codesourcery.com> wrote:

> On 05/09/18 19:07, Janne Blomqvist wrote:
> > The argument must be of type size_type_node, not sizetype. Please instead
> > use
> >
> > size = build_zero_cst (size_type_node);
> >
> >
> >>          * trans-intrinsic.c (conv_intrinsic_event_query): Convert
> computed
> >>          index to a size_t type.
> >>
> >
> > Using integer_type_node is wrong, but the correct type for calculating
> > array indices (lbound, ubound,  etc.) is not size_type_node but rather
> > gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
> > that, and then fold_convert index to size_type_node just before
> generating
> > the call to event_query.
> >
> >
> >>          * trans-stmt.c (gfc_trans_event_post_wait): Likewise.
> >>
> >
> > Same here as above.
>
> How is the attached? I retested and found no regressions.
>
> Andrew
>

Re: [PATCH 08/25] Fix co-array allocation

[Andrew Stubbs]

On 19/09/18 22:18, Damian Rouson wrote:
> Has this been tested in multi-image execution using OpenCoarrays?   If 
> not, I would be glad to assist with installing OpenCoarrays so that it 
> can be part of the testing process.

It's been tested with the GCC testsuite -- the same suite that found the 
issue in the first place.

If you want to port your tool to GCN that would be cool, but I suspect 
non-trivial.

> On a related note, two Sourcery Institute developers have attempted to 
> edit the GCC build system to make the downloading and building of 
> OpenCoarrays automatically part of the gfortran build process.  Neither 
> developer succeeded.  If anyone has any interest in figuring out how to 
> do this, it will prevent a lot of potential regressions when 
> single-image testing doesn't expose issues that only arise with 
> multi-image execution.

I suggest you post this question in a fresh thread.

Andrew

Re: [PATCH 08/25] Fix co-array allocation

[Damian Rouson]

On Wed, Sep 19, 2018 at 3:30 PM Andrew Stubbs <ams@codesourcery.com> wrote:

>
> If you want to port your tool to GCN that would be cool, but I suspect
> non-trivial.
>

To clarify, OpenCoarrays is not a tool.  It is the parallel ABI required to
create executable programs capable of executing in multiple images as
required by the Fortran 2008 standard.  Multi-image execution is the reason
coarray features exist so I hope the maintainers won't approve a patch that
impacts coarray features but has not been tested against OpenCoarrays,
which has its own test suite.  Again, I would be glad to assist with
installing OpenCoarrays on your system. Whether it's trivial or not, it's
essential to protect against breaking a large feature set that is part of
Fortran 2008 and 2018.

Damian

Re: [PATCH 08/25] Fix co-array allocation

[Janne Blomqvist]

On Wed, Sep 19, 2018 at 7:24 PM Andrew Stubbs <ams@codesourcery.com> wrote:

> On 05/09/18 19:07, Janne Blomqvist wrote:
> > The argument must be of type size_type_node, not sizetype. Please instead
> > use
> >
> > size = build_zero_cst (size_type_node);
> >
> >
> >>          * trans-intrinsic.c (conv_intrinsic_event_query): Convert
> computed
> >>          index to a size_t type.
> >>
> >
> > Using integer_type_node is wrong, but the correct type for calculating
> > array indices (lbound, ubound,  etc.) is not size_type_node but rather
> > gfc_array_index_type (which in practice maps to ptrdiff_t). So please use
> > that, and then fold_convert index to size_type_node just before
> generating
> > the call to event_query.
> >
> >
> >>          * trans-stmt.c (gfc_trans_event_post_wait): Likewise.
> >>
> >
> > Same here as above.
>
> How is the attached? I retested and found no regressions.
>
> Andrew
>

Ok, looks good.

There are some other remaining incorrect uses of integer_type_node (at
least one visible in the diff), but that can be done as a separate patch
(not saying you must do it as a precondition for anything, though it would
of course be nice if you would. :) )

-- 
Janne Blomqvist

Re: [PATCH 08/25] Fix co-array allocation

[Andrew Stubbs]

On 20/09/18 16:56, Janne Blomqvist wrote:
> Ok, looks good.

Thanks.

> There are some other remaining incorrect uses of integer_type_node (at 
> least one visible in the diff), but that can be done as a separate patch 
> (not saying you must do it as a precondition for anything, though it 
> would of course be nice if you would. :) )

I'm not confident I can tell what should be integer_type_node, and what 
should not?

Once it gets to build_call_expr_loc it's clear that the types should 
match the function signature, but the intermediate values' types are not 
obvious to me.

Andrew

Re: [PATCH 08/25] Fix co-array allocation

[Thomas Koenig]

Hi Damian,

> On a related note, two Sourcery Institute developers have attempted to edit
> the GCC build system to make the downloading and building of OpenCoarrays
> automatically part of the gfortran build process.  Neither developer
> succeeded.

We addressed integrating OpenCoarray into the gcc source tree at the
recent Gcc summit during the gfortran BoF session.

Feedback from people working for big Linux distributions was that they
would prefer to package OpenCoarrays as a separate library.
(They also mentioned it was quite hard to build.)

Maybe these people could use some help from you.

Regards

	Thomas

Re: [PATCH 08/25] Fix co-array allocation

[Damian Rouson]

On Thu, Sep 20, 2018 at 1:01 PM Thomas Koenig <tkoenig@netcologne.de> wrote:

>
> We addressed integrating OpenCoarray into the gcc source tree at the
> recent Gcc summit during the gfortran BoF session.
>

I agree with keeping it as a separate code base, but comments from some
gfortran developers on the gfortran mailing list suggest that they liked
the idea of integrating the building of OpenCoarrays into the GCC build
system to simplify multi-image testing.


> Feedback from people working for big Linux distributions was that they
> would prefer to package OpenCoarrays as a separate library.
> (They also mentioned it was quite hard to build.)
>
> Maybe these people could use some help from you.
>

Thanks for the feedback.  Please feel free to put me in touch with them or
suggest that they submit issues on the OpenCoarrays repository.  We would
be glad to help.  We've put a lot of time into addressing installation
issues that have been submitted to us and we'll continue to do so if we
receive reports.

Damian

Re: [PATCH 08/25] Fix co-array allocation

[Toon Moene]

On 09/20/2018 10:01 PM, Thomas Koenig wrote:

> Hi Damian,
> 
>> On a related note, two Sourcery Institute developers have attempted to 
>> edit
>> the GCC build system to make the downloading and building of OpenCoarrays
>> automatically part of the gfortran build process.  Neither developer
>> succeeded.
> 
> We addressed integrating OpenCoarray into the gcc source tree at the
> recent Gcc summit during the gfortran BoF session.
> 
> Feedback from people working for big Linux distributions was that they
> would prefer to package OpenCoarrays as a separate library.
> (They also mentioned it was quite hard to build.)

Well, Linux distributors have to fit the build of OpenCoarrays into 
*their* build system, which might be just as complicated as we trying it 
to force it into *gcc's* build system ...

For an individual, OpenCoarrays is not hard to build, and the web page 
www.opencoarrays.org offers multiple solutions:

"Installation via package management is generally the easiest and most 
reliable option.   See below for the package-management installation 
options for Linux, macOS, and FreeBSD.  Alternatively, download and 
build the latest OpenCoarrays release  via the contained installation 
scripts or with CMake."

I choose the cmake based one, because I already had cmake installed to 
be able to build ECMWF's (ecmwf.int) eccodes package. It probably helped 
that I also already had openmpi installed. From my command history:

  1754  tar zxvf ~/Downloads/OpenCoarrays-2.2.0.tar.gz
  1755  cd OpenCoarrays-2.2.0/
  1756  ls
  1757  less README.md
  1758  cd ..
  1759  mkdir opencoarrays-build
  1760  cd opencoarrays-build
  1761  (export FC=gfortran; export CC=gcc; cmake ../OpenCoarrays-2.2.0/ 
-DCMAKE_INSTALL_PREFIX=$HOME/opencoarrays)
  1762  make
  1763  make test
  1764  make install

After that, it was a breeze to test my mock weather program 
(moene.org/~toon/random-weather.f90), that I had built until then only 
with -fcoarray=single.

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/wiki/GFortran#news

OpenCoarrays integration with gfortran

[Jerry DeLisle]

My apologies for kidnapping this thread:
On 9/20/18 1:01 PM, Thomas Koenig wrote:
> Hi Damian,
> 
>> On a related note, two Sourcery Institute developers have attempted to 
>> edit
>> the GCC build system to make the downloading and building of OpenCoarrays
>> automatically part of the gfortran build process.  Neither developer
>> succeeded.
> 
> We addressed integrating OpenCoarray into the gcc source tree at the
> recent Gcc summit during the gfortran BoF session.
> 
> Feedback from people working for big Linux distributions was that they
> would prefer to package OpenCoarrays as a separate library.
> (They also mentioned it was quite hard to build.)

I would like to put in my humble 2 cents worth here.

OpenCoarrays was/is intended for a very broad audience, various large 
systems such as Cray, etc. I think this influenced heavily the path of 
its development, which is certainly OK.

It was/is intended to interface libraries such as OpenMPI or MPICH to 
gfortran as well as other Fortran compilers.

The actual library source code is contained mostly in one source file. 
After all the attempts to integrate into the GNU build systems without 
much success my thinking has shifted. Keep in mind that the OpenCoarrays 
implementation is quite dependent on gfortran and in fact has to do 
special things in the build dependent on the version of gcc/gfortran a 
user happens to use.  I dont think this is a good situation.

So I see two realistic strategies.  The first is already talked about a 
lot and is the cleanest approach for gfortran:

1) Focus on distribution packages such as Fedora, Debian, Ubuntu, 
Windows, etc. Building of these packages needs to be automated into the 
distributions. I think mostly this is what is happening and relies on 
the various distribution maintainers to do so.  Their support is greatly 
appreciated and this really is the cleanest approach.

The second option is not discussed as much because it leaves 
OpenCoarrays behind in a sense and requires an editing cycle in two 
places to fix bugs or add features.

2) Take the one source file, edit out all the macros that define 
prefixes to function calls, hard code the gfortran prefixes etc and fork 
it directly into the libgfortran library under GPL with attributions to 
the original developers as appropriate.

Strategy 2 would lock into specific current standard versions of the MPI 
interface and would support less bleeding edge changes.  It would also 
require either OpenMPI or MPICH as a new gfortran dependency for 
building, which not all users may need. So we would need some 
configuration magic to enable or disable this portion of the build. 
Something like --with-MPI-support would do the trick.

Strategy 2 does add burden to gfortran maintainers who are already 
overloaded. But, as the code matures the burden would decrease, 
particularly once TEAMS are finished.

Strategy 2 does have some advantages. For example, eliminating the need 
for separate CAF and CAFRUN scripts which are a wrapper on gfortran. 
The coarray features are part of the Fortran language and gfortran 
should just "handle it" transparently using an environment variable to 
define the number of images at run time. It would also actually 
eliminate the need to manage all of the separate distribution packages. 
So from a global point of view the overall maintanance effort would be 
reduced.

Strategy 2 would enable a set of users who are not focused so much on 
distributions and loading packages, etc etc and those who are dependent 
on getting through bureaucratic administrations who already are loading 
gfortran on systems and would not have to also get another package 
approved.  People would just have to stop thinking about it and just use it.

So I think there are real advantages to Strategy 2 as well as Strategy 1 
and think it should be at least included in discussions. I would even 
suggest there is likely a combination of 1 and 2 that may hit the mark. 
For example, keeping OpenCoarrays as a separate package for bleeding 
edge development and migrating the stable features into libgfortran on a 
less frequent cycle.

As I said, my 2 cents worth.

Regards to all,

Jerry

Re: OpenCoarrays integration with gfortran

[Janne Blomqvist]

On Fri, Sep 21, 2018 at 7:25 PM Jerry DeLisle <jvdelisle@charter.net> wrote:

> My apologies for kidnapping this thread:
> On 9/20/18 1:01 PM, Thomas Koenig wrote:
> > Hi Damian,
> >
> >> On a related note, two Sourcery Institute developers have attempted to
> >> edit
> >> the GCC build system to make the downloading and building of
> OpenCoarrays
> >> automatically part of the gfortran build process.  Neither developer
> >> succeeded.
> >
> > We addressed integrating OpenCoarray into the gcc source tree at the
> > recent Gcc summit during the gfortran BoF session.
> >
> > Feedback from people working for big Linux distributions was that they
> > would prefer to package OpenCoarrays as a separate library.
> > (They also mentioned it was quite hard to build.)
>
> I would like to put in my humble 2 cents worth here.
>
> OpenCoarrays was/is intended for a very broad audience, various large
> systems such as Cray, etc. I think this influenced heavily the path of
> its development, which is certainly OK.
>
> It was/is intended to interface libraries such as OpenMPI or MPICH to
> gfortran as well as other Fortran compilers.
>
> The actual library source code is contained mostly in one source file.
> After all the attempts to integrate into the GNU build systems without
> much success my thinking has shifted. Keep in mind that the OpenCoarrays
> implementation is quite dependent on gfortran and in fact has to do
> special things in the build dependent on the version of gcc/gfortran a
> user happens to use.  I dont think this is a good situation.
>
> So I see two realistic strategies.  The first is already talked about a
> lot and is the cleanest approach for gfortran:
>
> 1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
> Windows, etc. Building of these packages needs to be automated into the
> distributions. I think mostly this is what is happening and relies on
> the various distribution maintainers to do so.  Their support is greatly
> appreciated and this really is the cleanest approach.
>
> The second option is not discussed as much because it leaves
> OpenCoarrays behind in a sense and requires an editing cycle in two
> places to fix bugs or add features.
>
> 2) Take the one source file, edit out all the macros that define
> prefixes to function calls, hard code the gfortran prefixes etc and fork
> it directly into the libgfortran library under GPL with attributions to
> the original developers as appropriate.
>
> Strategy 2 would lock into specific current standard versions of the MPI
> interface and would support less bleeding edge changes.  It would also
> require either OpenMPI or MPICH as a new gfortran dependency for
> building, which not all users may need. So we would need some
> configuration magic to enable or disable this portion of the build.
> Something like --with-MPI-support would do the trick.
>
> Strategy 2 does add burden to gfortran maintainers who are already
> overloaded. But, as the code matures the burden would decrease,
> particularly once TEAMS are finished.
>
> Strategy 2 does have some advantages. For example, eliminating the need
> for separate CAF and CAFRUN scripts which are a wrapper on gfortran.
> The coarray features are part of the Fortran language and gfortran
> should just "handle it" transparently using an environment variable to
> define the number of images at run time. It would also actually
> eliminate the need to manage all of the separate distribution packages.
> So from a global point of view the overall maintanance effort would be
> reduced.
>
> Strategy 2 would enable a set of users who are not focused so much on
> distributions and loading packages, etc etc and those who are dependent
> on getting through bureaucratic administrations who already are loading
> gfortran on systems and would not have to also get another package
> approved.  People would just have to stop thinking about it and just use
> it.
>
> So I think there are real advantages to Strategy 2 as well as Strategy 1
> and think it should be at least included in discussions. I would even
> suggest there is likely a combination of 1 and 2 that may hit the mark.
> For example, keeping OpenCoarrays as a separate package for bleeding
> edge development and migrating the stable features into libgfortran on a
> less frequent cycle.
>
> As I said, my 2 cents worth.
>
> Regards to all,
>
> Jerry
>
>
I recall one motivation for the current sort-of loose coupling between the
coarray library and gfortran was to support, at runtime, different MPI
libraries.  This can be useful on cluster and supercomputers, where it's
important to use a MPI library that can use the high-performance cluster
network. If libgfortran includes the coarray library which links against a
MPI library, it means libgfortran has to be rebuilt against every MPI
library in use on a system, and most likely, one cannot use the
distro-provided gfortran.  This might not be insurmountable on cluster
using some kind of module system, but still.

I guess it might be possible to use weak symbols, like we currently use for
some things in libgfortran (e.g. clock_gettime), but that would mean a
quite big diff compared to upstream OpenCoarrays. And how to handle targets
that don't support weak symbols in some sane fashion, etc.

-- 
Janne Blomqvist

Re: OpenCoarrays integration with gfortran

[Richard Biener]

On September 21, 2018 6:24:45 PM GMT+02:00, Jerry DeLisle <jvdelisle@charter.net> wrote:
>My apologies for kidnapping this thread:
>On 9/20/18 1:01 PM, Thomas Koenig wrote:
>> Hi Damian,
>> 
>>> On a related note, two Sourcery Institute developers have attempted
>to 
>>> edit
>>> the GCC build system to make the downloading and building of
>OpenCoarrays
>>> automatically part of the gfortran build process.  Neither developer
>>> succeeded.
>> 
>> We addressed integrating OpenCoarray into the gcc source tree at the
>> recent Gcc summit during the gfortran BoF session.
>> 
>> Feedback from people working for big Linux distributions was that
>they
>> would prefer to package OpenCoarrays as a separate library.
>> (They also mentioned it was quite hard to build.)
>
>I would like to put in my humble 2 cents worth here.
>
>OpenCoarrays was/is intended for a very broad audience, various large 
>systems such as Cray, etc. I think this influenced heavily the path of 
>its development, which is certainly OK.
>
>It was/is intended to interface libraries such as OpenMPI or MPICH to 
>gfortran as well as other Fortran compilers.
>
>The actual library source code is contained mostly in one source file. 
>After all the attempts to integrate into the GNU build systems without 
>much success my thinking has shifted. Keep in mind that the
>OpenCoarrays 
>implementation is quite dependent on gfortran and in fact has to do 
>special things in the build dependent on the version of gcc/gfortran a 
>user happens to use.  I dont think this is a good situation.
>
>So I see two realistic strategies.  The first is already talked about a
>
>lot and is the cleanest approach for gfortran:
>
>1) Focus on distribution packages such as Fedora, Debian, Ubuntu, 
>Windows, etc. Building of these packages needs to be automated into the
>
>distributions. I think mostly this is what is happening and relies on 
>the various distribution maintainers to do so.  Their support is
>greatly 
>appreciated and this really is the cleanest approach.
>
>The second option is not discussed as much because it leaves 
>OpenCoarrays behind in a sense and requires an editing cycle in two 
>places to fix bugs or add features.
>
>2) Take the one source file, edit out all the macros that define 
>prefixes to function calls, hard code the gfortran prefixes etc and
>fork 
>it directly into the libgfortran library under GPL with attributions to
>
>the original developers as appropriate.
>
>Strategy 2 would lock into specific current standard versions of the
>MPI 
>interface and would support less bleeding edge changes.  It would also 
>require either OpenMPI or MPICH as a new gfortran dependency for 
>building, which not all users may need. So we would need some 
>configuration magic to enable or disable this portion of the build. 
>Something like --with-MPI-support would do the trick.
>
>Strategy 2 does add burden to gfortran maintainers who are already 
>overloaded. But, as the code matures the burden would decrease, 
>particularly once TEAMS are finished.
>
>Strategy 2 does have some advantages. For example, eliminating the need
>
>for separate CAF and CAFRUN scripts which are a wrapper on gfortran. 
>The coarray features are part of the Fortran language and gfortran 
>should just "handle it" transparently using an environment variable to 
>define the number of images at run time. It would also actually 
>eliminate the need to manage all of the separate distribution packages.
>
>So from a global point of view the overall maintanance effort would be 
>reduced.
>
>Strategy 2 would enable a set of users who are not focused so much on 
>distributions and loading packages, etc etc and those who are dependent
>
>on getting through bureaucratic administrations who already are loading
>
>gfortran on systems and would not have to also get another package 
>approved.  People would just have to stop thinking about it and just
>use it.
>
>So I think there are real advantages to Strategy 2 as well as Strategy
>1 
>and think it should be at least included in discussions. I would even 
>suggest there is likely a combination of 1 and 2 that may hit the mark.
>
>For example, keeping OpenCoarrays as a separate package for bleeding 
>edge development and migrating the stable features into libgfortran on
>a 
>less frequent cycle.

Sounds reasonable to me. License issues will be the most difficult here given integration with libgfortran likely requires a FSF copyright rather than just a compatible license. 

Richard. 

>As I said, my 2 cents worth.
>
>Regards to all,
>
>Jerry

Re: OpenCoarrays integration with gfortran

[Damian Rouson]

On Fri, Sep 21, 2018 at 9:25 AM Jerry DeLisle <jvdelisle@charter.net> wrote:

> The actual library source code is contained mostly in one source file.

There are as many files as there are options for the underlying
parallel programming
model.  The default is MPI, but I've co-authored conference papers last year
and this year in which the OpenCoarrays OpenSHEM option outperformed MPI.
One paper even described a platform on which OpenSHMEM was the only option
beyond a few thousand cores because the required MPI features were immature on
that platform.  Early versions of OpenCoarrays also provided GASNet
and ARMCI options.
I recommend against tying gfortran to MPI only.

> After all the attempts to integrate into the GNU build systems without
> much success my thinking has shifted.

Thanks for all your efforts!

> Keep in mind that the OpenCoarrays
> implementation is quite dependent on gfortran and in fact has to do
> special things in the build dependent on the version of gcc/gfortran a
> user happens to use.  I dont think this is a good situation.

I agree.  Possibly OpenCoarrays could drop support for older gfortran versions
at some point to avoid maintaining code that exists solely to support compiler
versions that are several years old.

>
> 1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
> Windows, etc. Building of these packages needs to be automated into the
> distributions.

This is the option that the OpenCoarrays documentation recommends as easiest for
most users.

> 2) Take the one source file, edit out all the macros that define
> prefixes to function calls, hard code the gfortran prefixes etc and fork
> it directly into the libgfortran library under GPL with attributions to
> the original developers as appropriate.

See above.   Also, this means that changes in the gfortran repository would not
propagate back upstream unless each gfortran developer agrees to
distribute his or her
work under both GPL and BSD.  Even that is only feasible if the copied
files stay cohesive
and don't reference code outside the copied file.  I think it's more
likely that copying the code
into gfortran would be a branch point, after which the relevant files
would diverge and
work on the GPL side would be harder to fund than the BSD side.

Most commercial entities are more likely to contribute to a
BSD-licensed project than a
GPL-licensed one.  Over the past several months, one commercial compiler vendor
authorized one of their developers to contribute to OpenCoarrays. and
another commercial
compiler vendor invited community input on whether to use OpenCoarrays
during a public
teleconference.  The prospect of commercial support is the motivation
for using BSD.

> Strategy 2 does have some advantages. For example, eliminating the need
> for separate CAF and CAFRUN scripts which are a wrapper on gfortran.

Even in the case of just one underlying parallel programming model,
this is tricky.  To wit, Cray uses
a compiler wrapper and a program launcher.  Intel was able to
eliminate the compiler wrapper,
but still required a program launcher for distributed-memory execution
until recently.  I don't
know the details, but I've heard it was not trivial for Intel to
accomplish this and I imagine it would be
even more complicated if they weren't hardwiring Intel MPI into their back-end.

> People would just have to stop thinking about it and just use it.

The same would be true if someone could coax the GCC build system to
build OpenCoarrays
just as it builds other prerequisites.  The big difference is that
OpenCoarrays is a prerequisite
for using gfortran rather than for building gfortran so it needs to be
built after gfortran rather
than before like other prerequisites.  The real problem is finding
anyone who can work the
proper magic in the GCC build system.

Thanks for your input.  I hope my response is helpful.

Damian

Re: OpenCoarrays integration with gfortran

[Jerry DeLisle]

On 9/21/18 1:16 PM, Damian Rouson wrote:> On Fri, Sep 21, 2018 at 9:25 
AM Jerry DeLisle <jvdelisle@charter.net> wrote:
 >
 >> The actual library source code is contained mostly in one source file.
 >
 > There are as many files as there are options for the underlying
 > parallel programming
 > model.  The default is MPI, but I've co-authored conference papers 
last year
 > and this year in which the OpenCoarrays OpenSHEM option outperformed MPI.
 > One paper even described a platform on which OpenSHMEM was the only 
option
 > beyond a few thousand cores because the required MPI features were 
immature on
 > that platform.  Early versions of OpenCoarrays also provided GASNet
 > and ARMCI options.
 > I recommend against tying gfortran to MPI only.

I agree with you on this point. Perhaps the Opencoarrays implementation 
should somehow do some runtime introspection to allow the library to 
sync to whatever is desired on a given system. The gfortran interface 
was designed to be generic. Implementation should be more dynamic in run 
time linking and abstracted in such a way that OpenCoarrays could be 
compiled stand alone and use something like "plugins" to allow post 
build the determination of what which interface to use.

I am by no means a software expert in these techniques, but they are 
becoming common practice in other areas, for example linux/Gnu kernel 
modules

 >
 >> After all the attempts to integrate into the GNU build systems without
 >> much success my thinking has shifted.
 >
 > Thanks for all your efforts!
 >
 >> Keep in mind that the OpenCoarrays
 >> implementation is quite dependent on gfortran and in fact has to do
 >> special things in the build dependent on the version of gcc/gfortran a
 >> user happens to use.  I dont think this is a good situation.
 >
 > I agree.  Possibly OpenCoarrays could drop support for older gfortran 
versions
 > at some point to avoid maintaining code that exists solely to support 
compiler
 > versions that are several years old.

See my comments above about pluggable modules.  Maybe libgfortran should 
have this pluggable interface and Opencoarrays provide the plugins. 
Think how useful it would be to be able to choose the backend at time of 
execution based on a simple envoronment variable set by the user.

 >
 >>
 >> 1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
 >> Windows, etc. Building of these packages needs to be automated into the
 >> distributions.
 >
 > This is the option that the OpenCoarrays documentation recommends as 
easiest for
 > most users.

Agree.

 >
 >> 2) Take the one source file, edit out all the macros that define
 >> prefixes to function calls, hard code the gfortran prefixes etc and fork
 >> it directly into the libgfortran library under GPL with attributions to
 >> the original developers as appropriate.
 >
 > See above.   Also, this means that changes in the gfortran repository 
would not
 > propagate back upstream unless each gfortran developer agrees to
 > distribute his or her
 > work under both GPL and BSD.  Even that is only feasible if the copied
 > files stay cohesive

The flip of this would be to have the OpenCorrays developers to agree to 
the GPL and release under both. The libgfortran license says:

"Under Section 7 of GPL version 3, you are granted additional
permissions described in the GCC Runtime Library Exception, version
3.1, as published by the Free Software Foundation."

Probably worth a fresh look.

 > and don't reference code outside the copied file.  I think it's more
 > likely that copying the code
 > into gfortran would be a branch point, after which the relevant files
 > would diverge and
 > work on the GPL side would be harder to fund than the BSD side.
 >
 > Most commercial entities are more likely to contribute to a
 > BSD-licensed project than a
 > GPL-licensed one.  Over the past several months, one commercial 
compiler vendor
 > authorized one of their developers to contribute to OpenCoarrays. and
 > another commercial
 > compiler vendor invited community input on whether to use OpenCoarrays
 > during a public
 > teleconference.  The prospect of commercial support is the motivation
 > for using BSD.

I really have no commercial interest. So I will not comment on GPL vs 
BSD other than referring to the multitude of FSF recommendations about 
why one should choose one of the FSF flavors rather than BSD.

 >
 >> Strategy 2 does have some advantages. For example, eliminating the need
 >> for separate CAF and CAFRUN scripts which are a wrapper on gfortran.
 >
 > Even in the case of just one underlying parallel programming model,
 > this is tricky.  To wit, Cray uses
 > a compiler wrapper and a program launcher.  Intel was able to
 > eliminate the compiler wrapper,
 > but still required a program launcher for distributed-memory execution
 > until recently.  I don't
 > know the details, but I've heard it was not trivial for Intel to
 > accomplish this and I imagine it would be
 > even more complicated if they weren't hardwiring Intel MPI into their 
back-end.

Well here is one commercial entity that did not shy away from 
'hardwiring' MPI, Regardless, using plugins would resolve concerns about 
which MPI to use or whether to use shared memory or some other model.

 >
 >> People would just have to stop thinking about it and just use it.
 >
 > The same would be true if someone could coax the GCC build system to
 > build OpenCoarrays
 > just as it builds other prerequisites.  The big difference is that
 > OpenCoarrays is a prerequisite
 > for using gfortran rather than for building gfortran so it needs to be
 > built after gfortran rather
 > than before like other prerequisites.  The real problem is finding
 > anyone who can work the
 > proper magic in the GCC build system.

I dont see this as the real problem. The forking idea would resolve this 
fairly easily.  Above, you mentioned concern about locking into MPI. Do 
the packaged versions of OpenCoarrays not lock into MPI, either OpenMPI 
or MPICH? I have not tried one yet since I am waiting for the Fedora one 
to hit the release.

If the tight coupling is needed maybe there ought to be a set of 
libraries or modules, one for each "backend". (Back to my pluggable 
modules concept)  The more I think about it the more I think this is the 
fundamental design issue.

 >
 > Thanks for your input.  I hope my response is helpful.
 >
 > Damian
 >

As always, best regards.

Jerry

Re: OpenCoarrays integration with gfortran

[Toon Moene]

On 09/22/2018 01:23 AM, Jerry DeLisle wrote:

> On 9/21/18 1:16 PM, Damian Rouson wrote:> On Fri, Sep 21, 2018 at 9:25 
> AM Jerry DeLisle <jvdelisle@charter.net> wrote:

>  >> 1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
>  >> Windows, etc. Building of these packages needs to be automated into the
>  >> distributions.
>  >
>  > This is the option that the OpenCoarrays documentation recommends as 
> easiest for
>  > most users.
> 
> Agree.

I just installed opencoarrays on my system at home (Debian Testing):

root@moene:~# apt-get install libcoarrays-openmpi-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
   libcaf-openmpi-3
The following NEW packages will be installed:
   libcaf-openmpi-3 libcoarrays-openmpi-dev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 107 kB of archives.
After this operation, 317 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ftp.nl.debian.org/debian testing/main amd64 
libcaf-openmpi-3 amd64 2.2.0-3 [38.2 kB]
Get:2 http://ftp.nl.debian.org/debian testing/main amd64 
libcoarrays-openmpi-dev amd64 2.2.0-3 [68.9 kB]
Fetched 107 kB in 0s (634 kB/s)
Selecting previously unselected package libcaf-openmpi-3:amd64.
(Reading database ... 212249 files and directories currently installed.)
Preparing to unpack .../libcaf-openmpi-3_2.2.0-3_amd64.deb ...
Unpacking libcaf-openmpi-3:amd64 (2.2.0-3) ...
Selecting previously unselected package libcoarrays-openmpi-dev:amd64.
Preparing to unpack .../libcoarrays-openmpi-dev_2.2.0-3_amd64.deb ...
Unpacking libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
Processing triggers for libc-bin (2.27-6) ...

[ previously this led to apt errors, but not now. ]

and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of 
the way:

toon@moene:~$ ls -ld *pen*
drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij

and recompiled my stuff:

gfortran -g -fbacktrace -fcoarray=lib random-weather.f90 
-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi

[ Yes, the location of the libs is quite experimental, but OK for the 
"Testing" variant of Debian ... ]

I couldn't find cafrun, but mpirun works just fine:

toon@moene:~/src$ echo ' &config /' | mpirun --oversubscribe --bind-to 
none -np 20 ./a.out
Decomposition information on image    7 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image    6 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   11 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   15 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image    1 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   13 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   12 is    4 *    5 slabs with   21 * 
   18 grid cells on this image.
Decomposition information on image   20 is    4 *    5 slabs with   21 * 
   18 grid cells on this image.
Decomposition information on image    9 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   14 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   16 is    4 *    5 slabs with   21 * 
   18 grid cells on this image.
Decomposition information on image   17 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   18 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image    2 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image    4 is    4 *    5 slabs with   21 * 
   18 grid cells on this image.
Decomposition information on image    5 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image    3 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image    8 is    4 *    5 slabs with   21 * 
   18 grid cells on this image.
Decomposition information on image   10 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.
Decomposition information on image   19 is    4 *    5 slabs with   23 * 
   18 grid cells on this image.

... etc. (see http://moene.org/~toon/random-weather.f90).

I presume other Linux distributors will follow shortly (this *is* Debian 
Testing, which can be a bit testy at times - but I do trust my main 
business at home on it for over 15 years now).

Kind regards,

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/wiki/GFortran#news

Re: [PATCH 08/25] Fix co-array allocation

[Janne Blomqvist]

On Fri, Sep 21, 2018 at 10:33 AM Toon Moene <toon@moene.org> wrote:

> On 09/20/2018 10:01 PM, Thomas Koenig wrote:
>
> > Hi Damian,
> >
> >> On a related note, two Sourcery Institute developers have attempted to
> >> edit
> >> the GCC build system to make the downloading and building of
> OpenCoarrays
> >> automatically part of the gfortran build process.  Neither developer
> >> succeeded.
> >
> > We addressed integrating OpenCoarray into the gcc source tree at the
> > recent Gcc summit during the gfortran BoF session.
> >
> > Feedback from people working for big Linux distributions was that they
> > would prefer to package OpenCoarrays as a separate library.
> > (They also mentioned it was quite hard to build.)
>
> Well, Linux distributors have to fit the build of OpenCoarrays into
> *their* build system, which might be just as complicated as we trying it
> to force it into *gcc's* build system ...
>
> For an individual, OpenCoarrays is not hard to build, and the web page
> www.opencoarrays.org offers multiple solutions:
>
> "Installation via package management is generally the easiest and most
> reliable option.   See below for the package-management installation
> options for Linux, macOS, and FreeBSD.  Alternatively, download and
> build the latest OpenCoarrays release  via the contained installation
> scripts or with CMake."
>
> I choose the cmake based one, because I already had cmake installed to
> be able to build ECMWF's (ecmwf.int) eccodes package. It probably helped
> that I also already had openmpi installed. From my command history:
>
>   1754  tar zxvf ~/Downloads/OpenCoarrays-2.2.0.tar.gz
>   1755  cd OpenCoarrays-2.2.0/
>   1756  ls
>   1757  less README.md
>   1758  cd ..
>   1759  mkdir opencoarrays-build
>   1760  cd opencoarrays-build
>   1761  (export FC=gfortran; export CC=gcc; cmake ../OpenCoarrays-2.2.0/
> -DCMAKE_INSTALL_PREFIX=$HOME/opencoarrays)
>   1762  make
>   1763  make test
>   1764  make install
>

FWIW, this didn't work for me, as I want to use my own build of gfortran
trunk. It did correctly use the correct gfortran binary as specified by the
FC env. variable, but it still insists on linking against libgfortran.so.4
(installed by the system package manager) and not the libgfortran.so.5 from
my own gfortran installation (found both on LD_RUN_PATH and
LD_LIBRARY_PATH).  I tried -DCMAKE_PREFIX_PATH=... but that didn't work any
better. Gah, I hate cmake..

Any ideas?

-- 
Janne Blomqvist

Re: OpenCoarrays integration with gfortran

[Bernhard Reutner-Fischer]

On 23 September 2018 11:46:57 CEST, Toon Moene <toon@moene.org> wrote:
>On 09/22/2018 01:23 AM, Jerry DeLisle wrote:
>
>> On 9/21/18 1:16 PM, Damian Rouson wrote:> On Fri, Sep 21, 2018 at
>9:25 
>> AM Jerry DeLisle <jvdelisle@charter.net> wrote:
>
>>  >> 1) Focus on distribution packages such as Fedora, Debian, Ubuntu,
>>  >> Windows, etc. Building of these packages needs to be automated
>into the
>>  >> distributions.
>>  >
>>  > This is the option that the OpenCoarrays documentation recommends
>as 
>> easiest for
>>  > most users.
>> 
>> Agree.
>
>I just installed opencoarrays on my system at home (Debian Testing):
>
>root@moene:~# apt-get install libcoarrays-openmpi-dev
>Reading package lists... Done
>Building dependency tree
>Reading state information... Done
>The following additional packages will be installed:
>   libcaf-openmpi-3
>The following NEW packages will be installed:
>   libcaf-openmpi-3 libcoarrays-openmpi-dev
>0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
>Need to get 107 kB of archives.
>After this operation, 317 kB of additional disk space will be used.
>Do you want to continue? [Y/n]
>Get:1 http://ftp.nl.debian.org/debian testing/main amd64 
>libcaf-openmpi-3 amd64 2.2.0-3 [38.2 kB]
>Get:2 http://ftp.nl.debian.org/debian testing/main amd64 
>libcoarrays-openmpi-dev amd64 2.2.0-3 [68.9 kB]
>Fetched 107 kB in 0s (634 kB/s)
>Selecting previously unselected package libcaf-openmpi-3:amd64.
>(Reading database ... 212249 files and directories currently
>installed.)
>Preparing to unpack .../libcaf-openmpi-3_2.2.0-3_amd64.deb ...
>Unpacking libcaf-openmpi-3:amd64 (2.2.0-3) ...
>Selecting previously unselected package libcoarrays-openmpi-dev:amd64.
>Preparing to unpack .../libcoarrays-openmpi-dev_2.2.0-3_amd64.deb ...
>Unpacking libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
>Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
>Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
>Processing triggers for libc-bin (2.27-6) ...
>
>[ previously this led to apt errors, but not now. ]
>
>and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of 
>the way:
>
>toon@moene:~$ ls -ld *pen*
>drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
>drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
>drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
>
>and recompiled my stuff:
>
>gfortran -g -fbacktrace -fcoarray=lib random-weather.f90 
>-L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
>
>[ Yes, the location of the libs is quite experimental, but OK for the 
>"Testing" variant of Debian ... ]

Are you sure you need the -L?
For me a simple  -fcoarray=lib -lcaf_mpi
links fine.
Along the same lines a simple
$ mpirun -np 4 ./a.out
runs fine as expected, like any other mpi program.

Cheers,
>
>I couldn't find cafrun, but mpirun works just fine:
>
>toon@moene:~/src$ echo ' &config /' | mpirun --oversubscribe --bind-to 
>none -np 20 ./a.out
>Decomposition information on image    7 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image    6 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   11 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   15 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image    1 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   13 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   12 is    4 *    5 slabs with   21
>* 
>   18 grid cells on this image.
>Decomposition information on image   20 is    4 *    5 slabs with   21
>* 
>   18 grid cells on this image.
>Decomposition information on image    9 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   14 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   16 is    4 *    5 slabs with   21
>* 
>   18 grid cells on this image.
>Decomposition information on image   17 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   18 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image    2 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image    4 is    4 *    5 slabs with   21
>* 
>   18 grid cells on this image.
>Decomposition information on image    5 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image    3 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image    8 is    4 *    5 slabs with   21
>* 
>   18 grid cells on this image.
>Decomposition information on image   10 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>Decomposition information on image   19 is    4 *    5 slabs with   23
>* 
>   18 grid cells on this image.
>
>... etc. (see http://moene.org/~toon/random-weather.f90).
>
>I presume other Linux distributors will follow shortly (this *is*
>Debian 
>Testing, which can be a bit testy at times - but I do trust my main 
>business at home on it for over 15 years now).
>
>Kind regards,

Re: OpenCoarrays integration with gfortran

[Toon Moene]

[ dropping mailing list gcc-patches, as it is not relevant to *this* 
discussion :-) ]

On 09/23/2018 06:47 PM, Bernhard Reutner-Fischer wrote:

> On 23 September 2018 11:46:57 CEST, Toon Moene <toon@moene.org> wrote:

>> and recompiled my stuff:
>>
>> gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
>> -L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
>>
>> [ Yes, the location of the libs is quite experimental, but OK for the
>> "Testing" variant of Debian ... ]
> 
> Are you sure you need the -L?
> For me a simple  -fcoarray=lib -lcaf_mpi
> links fine.

I get this:

toon@moene:~/src$ gfortran -g -fbacktrace -fcoarray=lib 
random-weather.f90 -lcaf_mpi
/usr/bin/ld: cannot find -lcaf_mpi
collect2: error: ld returned 1 exit status

Are you sure the linker isn't finding another libcaf_mpi.so ?

> Along the same lines a simple
> $ mpirun -np 4 ./a.out
> runs fine as expected, like any other mpi program.

Yeah, sorry. I was already using the options  --oversubscribe --bind-to 
none (of which the  --oversubscribe certainly is necessary) before I 
installed Debian's coarray packages. I need it because otherwise openmpi 
will complain that I only have 10 cores (which is correct).

Kind regards,

-- 
Toon Moene - e-mail: toon@moene.org - phone: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
At home: http://moene.org/~toon/; weather: http://moene.org/~hirlam/
Progress of GNU Fortran: http://gcc.gnu.org/wiki/GFortran#news

Re: OpenCoarrays integration with gfortran

[Bernhard Reutner-Fischer]

On Sun, 23 Sep 2018 at 21:17, Toon Moene <toon@moene.org> wrote:
>
> [ dropping mailing list gcc-patches, as it is not relevant to *this*
> discussion :-) ]

yea, sorry for that.
>
> On 09/23/2018 06:47 PM, Bernhard Reutner-Fischer wrote:
>
> > On 23 September 2018 11:46:57 CEST, Toon Moene <toon@moene.org> wrote:
>
> >> and recompiled my stuff:
> >>
> >> gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
> >> -L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
> >>
> >> [ Yes, the location of the libs is quite experimental, but OK for the
> >> "Testing" variant of Debian ... ]
> >
> > Are you sure you need the -L?
> > For me a simple  -fcoarray=lib -lcaf_mpi
> > links fine.
>
> I get this:
>
> toon@moene:~/src$ gfortran -g -fbacktrace -fcoarray=lib
> random-weather.f90 -lcaf_mpi
> /usr/bin/ld: cannot find -lcaf_mpi
> collect2: error: ld returned 1 exit status
>
> Are you sure the linker isn't finding another libcaf_mpi.so ?

Yes, i'm certain it finds the right libcaf_mpi and openmpi, but then
you're on testing and i'm on sid, maybe that's a difference.
For reference (my compiler built from trunk lives in /opt/$(uname
-m)/gcc-9.0.mine):
$ cat ~/tmp/hello.f08;echo EOF
implicit none
character(len=64) :: str, hi[*]
integer :: i, nprocs
nprocs = num_images()
do i=1, nprocs
  !write(hi[this_image()], '(i0,a,i0,": ",a)') this_image(), '/',
nprocs, 'hello world' ! :) but data-ref :(
  write(str, '(i0,a,i0,": ",a)') this_image(), '/', nprocs, 'hello world' ! :(
  hi[this_image()] = str
end do
sync all
write(*,*) hi[this_image()]
end
EOF
$ gfortran -o hi -fcoarray=lib -lcaf_mpi ~/tmp/hello.f08 && mpirun -np 3 ./hi
 1/3: hello world
 2/3: hello world
 3/3: hello world
$ ldd ./hi
linux-vdso.so.1 (0x00007ffd03986000)
libcaf_mpi.so.3 => /usr/lib/x86_64-linux-gnu/libcaf_mpi.so.3
(0x00007fe9e6a72000)
libgfortran.so.5 => /opt/x86_64/gcc-9.0.mine/lib64/libgfortran.so.5
(0x00007fe9e677a000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe9e65e6000)
libgcc_s.so.1 => /opt/x86_64/gcc-9.0.mine/lib64/libgcc_s.so.1
(0x00007fe9e65c2000)
libquadmath.so.0 => /opt/x86_64/gcc-9.0.mine/lib64/libquadmath.so.0
(0x00007fe9e6576000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe9e63b9000)
libmpi.so.40 => /usr/lib/x86_64-linux-gnu/libmpi.so.40 (0x00007fe9e62af000)
libmpi_usempif08.so.40 =>
/usr/lib/x86_64-linux-gnu/libmpi_usempif08.so.40 (0x00007fe9e6278000)
libmpi_usempi_ignore_tkr.so.40 =>
/usr/lib/x86_64-linux-gnu/libmpi_usempi_ignore_tkr.so.40
(0x00007fe9e626c000)
libmpi_mpifh.so.40 => /usr/lib/x86_64-linux-gnu/libmpi_mpifh.so.40
(0x00007fe9e620f000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe9e6acb000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe9e620a000)
libopen-rte.so.40 => /usr/lib/x86_64-linux-gnu/libopen-rte.so.40
(0x00007fe9e6150000)
libopen-pal.so.40 => /usr/lib/x86_64-linux-gnu/libopen-pal.so.40
(0x00007fe9e60a3000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe9e6099000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fe9e6094000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe9e5e76000)
libhwloc.so.5 => /usr/lib/x86_64-linux-gnu/libhwloc.so.5 (0x00007fe9e5e35000)
libevent-2.1.so.6 => /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
(0x00007fe9e5bdd000)
libevent_pthreads-2.1.so.6 =>
/usr/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.6
(0x00007fe9e59da000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe9e59b9000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007fe9e59ab000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007fe9e59a0000)
$ dpkg -S /usr/lib/x86_64-linux-gnu/libcaf_mpi.so.3
libcaf-mpi3:amd64: /usr/lib/x86_64-linux-gnu/libcaf_mpi.so.3
$ dpkg -l libcaf-mpi3
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                  Version         Architecture    Description
+++-=====================-===============-===============-===============================================
ii  libcaf-mpi3:amd64     2.2.0-1         amd64           Co-Array
Fortran libraries for gfortran

I'd check if you really got the debian files so you don't run afoul a
manual install or leftovers thereof.
If it's really unpleasant in testing, it will at least ripple down
from sid sometimes and hence should work without hazzle soonish :)

HTH and cheers,
Bernhard

Re: OpenCoarrays integration with gfortran

[Alastair McKinstry]

On 23/09/2018 10:46, Toon Moene wrote:
> On 09/22/2018 01:23 AM, Jerry DeLisle wrote:
>
> I just installed opencoarrays on my system at home (Debian Testing):
>
> root@moene:~# apt-get install libcoarrays-openmpi-dev
> ...
> Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
> Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
> Processing triggers for libc-bin (2.27-6) ...
>
> [ previously this led to apt errors, but not now. ]
>
> and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of 
> the way:
>
> toon@moene:~$ ls -ld *pen*
> drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
> drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
> drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
>
> and recompiled my stuff:
>
> gfortran -g -fbacktrace -fcoarray=lib random-weather.f90 
> -L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
>
> [ Yes, the location of the libs is quite experimental, but OK for the 
> "Testing" variant of Debian ... ]
>
> I couldn't find cafrun, but mpirun works just fine:
>
> toon@moene:~/src$ echo ' &config /' | mpirun --oversubscribe --bind-to 
> none -np 20 ./a.out
> Decomposition information on image    7 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image    6 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   11 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   15 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image    1 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   13 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   12 is    4 *    5 slabs with   21 
> *   18 grid cells on this image.
> Decomposition information on image   20 is    4 *    5 slabs with   21 
> *   18 grid cells on this image.
> Decomposition information on image    9 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   14 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   16 is    4 *    5 slabs with   21 
> *   18 grid cells on this image.
> Decomposition information on image   17 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   18 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image    2 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image    4 is    4 *    5 slabs with   21 
> *   18 grid cells on this image.
> Decomposition information on image    5 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image    3 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image    8 is    4 *    5 slabs with   21 
> *   18 grid cells on this image.
> Decomposition information on image   10 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
> Decomposition information on image   19 is    4 *    5 slabs with   23 
> *   18 grid cells on this image.
>
> ... etc. (see http://moene.org/~toon/random-weather.f90).
>
> I presume other Linux distributors will follow shortly (this *is* 
> Debian Testing, which can be a bit testy at times - but I do trust my 
> main business at home on it for over 15 years now).
>
> Kind regards,
>
Thanks, good to see it being tested (I'm the Debian/Ubuntu packager).

caf /cafrun has been dropped (for the moment ? ) in favour of mpirun, 
but I've added pkg-config caf packages so that becomes an option.

    $ pkg-config caf-mpich --libs

    -L/usr/lib/x86_64-linux-gnu/open-coarrays/mpich/lib -lcaf_mpich -Wl,-z,relro -lmpich -lm -lbacktrace -lpthread -lrt

(My thinking is that for libraries in particular, the user need not know 
whether CAF is being used, and if lib foobar uses CAF, then adding a:

     Requires: caf

into the pkg-config file gives you the correct linking transparently.

The "strange" paths are due to Debians multiarch : it is possible to 
include libraries for multiple architectures simultaneously. This works 
ok with pkg-config and cmake , etc (which allow you to set 
PKG_CONFIG_PATH and have multiple pkgconfig files for different libs 
simultaneously) , but currently break wrappers such as caf / cafrun.

I can add a new package for caf / cafrun but would rather not. (W e 
currently don't do non-MPI CAF builds).

There is currently pkg-config files 'caf-mpich' and 'caf-openmpi' for 
testing, and I'm adding a default alias caf -> caf-$(default-MPI)

regards

Alastair




-- 
Alastair McKinstry, <alastair@sceal.ie>, <mckinstry@debian.org>, https://diaspora.sceal.ie/u/amckinstry
Misentropy: doubting that the Universe is becoming more disordered.

Re: OpenCoarrays integration with gfortran

[Richard Biener]

On Mon, Sep 24, 2018 at 12:58 PM Alastair McKinstry
<mckinstry@debian.org> wrote:
>
>
> On 23/09/2018 10:46, Toon Moene wrote:
> > On 09/22/2018 01:23 AM, Jerry DeLisle wrote:
> >
> > I just installed opencoarrays on my system at home (Debian Testing):
> >
> > root@moene:~# apt-get install libcoarrays-openmpi-dev
> > ...
> > Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
> > Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
> > Processing triggers for libc-bin (2.27-6) ...
> >
> > [ previously this led to apt errors, but not now. ]
> >
> > and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of
> > the way:
> >
> > toon@moene:~$ ls -ld *pen*
> > drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
> > drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
> > drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
> >
> > and recompiled my stuff:
> >
> > gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
> > -L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
> >
> > [ Yes, the location of the libs is quite experimental, but OK for the
> > "Testing" variant of Debian ... ]
> >
> > I couldn't find cafrun, but mpirun works just fine:
> >
> > toon@moene:~/src$ echo ' &config /' | mpirun --oversubscribe --bind-to
> > none -np 20 ./a.out
> > Decomposition information on image    7 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image    6 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   11 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   15 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image    1 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   13 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   12 is    4 *    5 slabs with   21
> > *   18 grid cells on this image.
> > Decomposition information on image   20 is    4 *    5 slabs with   21
> > *   18 grid cells on this image.
> > Decomposition information on image    9 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   14 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   16 is    4 *    5 slabs with   21
> > *   18 grid cells on this image.
> > Decomposition information on image   17 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   18 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image    2 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image    4 is    4 *    5 slabs with   21
> > *   18 grid cells on this image.
> > Decomposition information on image    5 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image    3 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image    8 is    4 *    5 slabs with   21
> > *   18 grid cells on this image.
> > Decomposition information on image   10 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> > Decomposition information on image   19 is    4 *    5 slabs with   23
> > *   18 grid cells on this image.
> >
> > ... etc. (see http://moene.org/~toon/random-weather.f90).
> >
> > I presume other Linux distributors will follow shortly (this *is*
> > Debian Testing, which can be a bit testy at times - but I do trust my
> > main business at home on it for over 15 years now).
> >
> > Kind regards,
> >
> Thanks, good to see it being tested (I'm the Debian/Ubuntu packager).
>
> caf /cafrun has been dropped (for the moment ? ) in favour of mpirun,
> but I've added pkg-config caf packages so that becomes an option.
>
>     $ pkg-config caf-mpich --libs
>
>     -L/usr/lib/x86_64-linux-gnu/open-coarrays/mpich/lib -lcaf_mpich -Wl,-z,relro -lmpich -lm -lbacktrace -lpthread -lrt
>
> (My thinking is that for libraries in particular, the user need not know
> whether CAF is being used, and if lib foobar uses CAF, then adding a:
>
>      Requires: caf
>
> into the pkg-config file gives you the correct linking transparently.
>
> The "strange" paths are due to Debians multiarch : it is possible to
> include libraries for multiple architectures simultaneously. This works
> ok with pkg-config and cmake , etc (which allow you to set
> PKG_CONFIG_PATH and have multiple pkgconfig files for different libs
> simultaneously) , but currently break wrappers such as caf / cafrun.
>
> I can add a new package for caf / cafrun but would rather not. (W e
> currently don't do non-MPI CAF builds).
>
> There is currently pkg-config files 'caf-mpich' and 'caf-openmpi' for
> testing, and I'm adding a default alias caf -> caf-$(default-MPI)

So I've tried packaging of OpenCoarrays for SUSE and noticed a few things:

 - caf by default links libcaf_mpi static (why?)
 - the build system makes the libcaf_mpi SONAME dependent on the compiler
   version(?), I once got libcaf_mpi2 and once libcaf_mpi3 (gcc7 vs. gcc8)

different SONAMEs definitely makes packaging difficult.  Of course since
there's the first point I may very well elide the shared library
alltogether....?

Other than that it seems to "work" (OBS home:rguenther/OpenCoarrays).

Richard.

> regards
>
> Alastair
>
>
>
>
> --
> Alastair McKinstry, <alastair@sceal.ie>, <mckinstry@debian.org>, https://diaspora.sceal.ie/u/amckinstry
> Misentropy: doubting that the Universe is becoming more disordered.
>

Re: OpenCoarrays integration with gfortran

[Jorge D'Elia]

----- Mensaje original -----
> De: "Richard Biener" <richard.guenther@gmail.com>
> Para: mckinstry@debian.org
> CC: "Toon Moene" <toon@moene.org>, "Jerry DeLisle" <jvdelisle@charter.net>, "Damian Rouson"
> <damian@sourceryinstitute.org>, "Thomas Koenig" <tkoenig@netcologne.de>, "Stubbs, Andrew" <ams@codesourcery.com>,
> "Janne Blomqvist" <blomqvist.janne@gmail.com>, "GCC Patches" <gcc-patches@gcc.gnu.org>, fortran@gcc.gnu.org
> Enviados: Jueves, 27 de Septiembre 2018 9:28:47
> Asunto: Re: OpenCoarrays integration with gfortran

> On Mon, Sep 24, 2018 at 12:58 PM Alastair McKinstry
> <mckinstry@debian.org> wrote:
>>
>>
>> On 23/09/2018 10:46, Toon Moene wrote:
>> > On 09/22/2018 01:23 AM, Jerry DeLisle wrote:
>> >
>> > I just installed opencoarrays on my system at home (Debian Testing):
>> >
>> > root@moene:~# apt-get install libcoarrays-openmpi-dev
>> > ...
>> > Setting up libcaf-openmpi-3:amd64 (2.2.0-3) ...
>> > Setting up libcoarrays-openmpi-dev:amd64 (2.2.0-3) ...
>> > Processing triggers for libc-bin (2.27-6) ...
>> >
>> > [ previously this led to apt errors, but not now. ]
>> >
>> > and moved my own installation of the OpenCoarrays-2.2.0.tar.gz out of
>> > the way:
>> >
>> > toon@moene:~$ ls -ld *pen*
>> > drwxr-xr-x 6 toon toon 4096 Aug 10 16:01 OpenCoarrays-2.2.0.opzij
>> > drwxr-xr-x 8 toon toon 4096 Sep 15 11:26 opencoarrays-build.opzij
>> > drwxr-xr-x 6 toon toon 4096 Sep 15 11:26 opencoarrays.opzij
>> >
>> > and recompiled my stuff:
>> >
>> > gfortran -g -fbacktrace -fcoarray=lib random-weather.f90
>> > -L/usr/lib/x86_64-linux-gnu/open-coarrays/openmpi/lib -lcaf_mpi
>> >
>> > [ Yes, the location of the libs is quite experimental, but OK for the
>> > "Testing" variant of Debian ... ]
>> >
>> > I couldn't find cafrun, but mpirun works just fine:
>> >
>> > toon@moene:~/src$ echo ' &config /' | mpirun --oversubscribe --bind-to
>> > none -np 20 ./a.out
>> > Decomposition information on image    7 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image    6 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   11 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   15 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image    1 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   13 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   12 is    4 *    5 slabs with   21
>> > *   18 grid cells on this image.
>> > Decomposition information on image   20 is    4 *    5 slabs with   21
>> > *   18 grid cells on this image.
>> > Decomposition information on image    9 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   14 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   16 is    4 *    5 slabs with   21
>> > *   18 grid cells on this image.
>> > Decomposition information on image   17 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   18 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image    2 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image    4 is    4 *    5 slabs with   21
>> > *   18 grid cells on this image.
>> > Decomposition information on image    5 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image    3 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image    8 is    4 *    5 slabs with   21
>> > *   18 grid cells on this image.
>> > Decomposition information on image   10 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> > Decomposition information on image   19 is    4 *    5 slabs with   23
>> > *   18 grid cells on this image.
>> >
>> > ... etc. (see http://moene.org/~toon/random-weather.f90).
>> >
>> > I presume other Linux distributors will follow shortly (this *is*
>> > Debian Testing, which can be a bit testy at times - but I do trust my
>> > main business at home on it for over 15 years now).
>> >
>> > Kind regards,
>> >
>> Thanks, good to see it being tested (I'm the Debian/Ubuntu packager).
>>
>> caf /cafrun has been dropped (for the moment ? ) in favour of mpirun,
>> but I've added pkg-config caf packages so that becomes an option.
>>
>>     $ pkg-config caf-mpich --libs
>>
>>     -L/usr/lib/x86_64-linux-gnu/open-coarrays/mpich/lib -lcaf_mpich -Wl,-z,relro
>>     -lmpich -lm -lbacktrace -lpthread -lrt
>>
>> (My thinking is that for libraries in particular, the user need not know
>> whether CAF is being used, and if lib foobar uses CAF, then adding a:
>>
>>      Requires: caf
>>
>> into the pkg-config file gives you the correct linking transparently.
>>
>> The "strange" paths are due to Debians multiarch : it is possible to
>> include libraries for multiple architectures simultaneously. This works
>> ok with pkg-config and cmake , etc (which allow you to set
>> PKG_CONFIG_PATH and have multiple pkgconfig files for different libs
>> simultaneously) , but currently break wrappers such as caf / cafrun.
>>
>> I can add a new package for caf / cafrun but would rather not. (W e
>> currently don't do non-MPI CAF builds).
>>
>> There is currently pkg-config files 'caf-mpich' and 'caf-openmpi' for
>> testing, and I'm adding a default alias caf -> caf-$(default-MPI)
> 
> So I've tried packaging of OpenCoarrays for SUSE and noticed a few things:
> 
> - caf by default links libcaf_mpi static (why?)
> - the build system makes the libcaf_mpi SONAME dependent on the compiler
>   version(?), I once got libcaf_mpi2 and once libcaf_mpi3 (gcc7 vs. gcc8)
> 
> different SONAMEs definitely makes packaging difficult.  Of course since
> there's the first point I may very well elide the shared library
> alltogether....?
> 
> Other than that it seems to "work" (OBS home:rguenther/OpenCoarrays).
> 
> Richard.

The issue of the static libcaf_mpi reminded me of an old email 
(from Lisandro Dalcin and me) about the coarray and Gfortran 
integration strategies. I just hit one part below (possibly 
something outdated). Maybe I can contribute something.

Regards,
Jorge.

#begin
About an MPI based library for the coarray model:

Nowadays, the coarray model is included in the lastest Fortran 
2008 standard. However, since several Fortran compilers are built 
on the basis of the C/C++ ones, there is not a native way to add 
the coarray model.

One option is to build an MPI based library for supporting 
the coarray model. In this case, it should be ensured that 
the final library is as neutral as possible on regards to 
a final user.

However, due to the fact that the MPI binaries are not yet 
compatible among the available MPI implementations, this 
issue should be overcomed somehow.

Among other possibilities, the following alternatives can be 
considered, from the simplest to the more elaborated ones:

(1) A static library libcaf_mpi.a which uses a specific MPI 
implementation when the library is built (e.g. OpenMPI or MPICH). 
However, it only works with the distribution available when the
library is built.

(2) A dynamic library libcaf_mpi.so that uses a specific MPI 
implementation when the library is built. However, again, it 
only works with the MPI implementation available when the library 
is built. From a practical point of view, this option is not very 
different than the previous one.

(3) A symbolic link libcaf_mpi.so that points to dynamic libraries 
libcaf_mpich.so or libcaf_openmpi.so. The sysadmin can manage the 
link using system tools as alternatives. The GNU/Linux distributions 
usually manage this infrastructure by themselves without additional 
work required from the compiler side. However, regular (non-root) 
users cannot switch the backend MPI. This is only available on POSIX 
systems.

(4) Different dynamic libraries named libcaf_mpi.so built for each 
MPI implementation, each of them installed in different directories, e.g. 
/mpich/libcaf_mpi.so, or /openmpi/libcaf_mpi.so.
By using the modules tool, users can select the preferred MPI 
implementation, e.g. module load mpich2-x86_64 in Fedora.
This works by adding entries in the LD_LIBRARY_PATH environment variable.
The GNU/Linux distributions usually manage this infrastructure by 
themselves and do not require additional work from the Fortran compiler 
side. Regular users are able to choose the preferred MPI implementation, 
and the dynamic linker loads the appropriate libcaf_mpi.so.

(5) A dynamic library  libcaf_mpi.so built by the dlopen() tool. This 
option could be practical if the number of MPI functions were not too 
large, and it can be built on both GNU/Linux and Windows OS.

(6) A dynamic library libcaf_mpi.so that is not linked with the MPI 
library, but uses the dlopen() and dlsym() to load and access the 
contents of a specific "MPI-linked" dynamic library, e.g.
libcaf_mpi_mpich2-openmpi-other.so. In this way, libcaf_mpi.so does 
not depend on any MPI implementation, but acts as a thin wrapper to 
the specific CAF+MPI library. By using environment variables or rc 
configuration files, the user can choose the preferred library to 
open at runtime with dlopen() in POSIX systems, or similar mechanisms
on Windows OS.
#end