Re: [committed] openmp: Add support for HBW or large capacity or interleaved memory through the libmemkind.so library

[Jakub Jelinek <jakub@redhat.com>]

On Thu, Jun 09, 2022 at 12:11:28PM +0200, Thomas Schwinge wrote:
> On 2022-06-09T10:19:03+0200, Jakub Jelinek via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
> > This patch adds support for dlopening libmemkind.so
> 
> Instead of 'dlopen'ing literally 'libmemkind.so':
> 
> > --- libgomp/allocator.c.jj    2022-06-08 08:21:03.099446883 +0200
> > +++ libgomp/allocator.c       2022-06-08 13:41:45.647133610 +0200
> 
> > +  void *handle = dlopen ("libmemkind.so", RTLD_LAZY);
> 
> ..., shouldn't this instead 'dlopen' 'libmemkind.so.0'?  At least for
> Debian/Ubuntu, the latter ('libmemkind.so.0') is shipped in the "library"
> package:

I agree and I've actually noticed it too right before committing, but I thought
I'll investigate and tweak incrementally because "libmemkind.so"
is what I've actually tested (it is what llvm libomp uses).

> 
>     $ apt-file list libmemkind0 | grep -F libmemkind.so
>     libmemkind0: /usr/lib/x86_64-linux-gnu/libmemkind.so.0
>     libmemkind0: /usr/lib/x86_64-linux-gnu/libmemkind.so.0.0.1
> 
> ..., but the former ('libmemkind.so') only in the "development" package:
> 
>     $ apt-file list libmemkind-dev | grep -F libmemkind.so
>     libmemkind-dev: /usr/lib/x86_64-linux-gnu/libmemkind.so
> 
> ..., which users of GCC/libgomp shouldn't have to care about.

Similarly in Fedora memkind package provides just
/usr/lib64/libautohbw.so.0
/usr/lib64/libautohbw.so.0.0.0
/usr/lib64/libmemkind.so.0
/usr/lib64/libmemkind.so.0.0.1
/usr/lib64/libmemtier.so.0
/usr/lib64/libmemtier.so.0.0.0
and
/usr/lib64/libautohbw.so
/usr/lib64/libmemkind.so
/usr/lib64/libmemtier.so
comes from memkind-devel.

> Any plans about test cases for this?  (Not trivial, I suppose?)

That is the hard part.
All the testing I've done so far were for atv_interleaved:
#include <omp.h>

int
main ()
{
  omp_alloctrait_t traits[3]
    = { { omp_atk_alignment, 64 },
        { omp_atk_fallback, omp_atv_null_fb },
        { omp_atk_partition, omp_atv_interleaved } };
  omp_allocator_handle_t a;

  a = omp_init_allocator (omp_default_mem_space, 3, traits);
  if (a == omp_null_allocator)
    return 1;
  void *p = omp_alloc (128, a);
  if (!p)
    return 2;
  void *q = omp_realloc (p, 256, a, a);
  if (!q)
    return 3;
  void *r = omp_calloc (1, 512, a);
  if (!r)
    return 4;
  omp_free (q, a);
  omp_free (r, a);
  return 0;
}
because that is something that works even on my devel WS, though
in the testcase one doesn't figure out if memkind was actually available and
whether the memory was indeed interleaved or not, just that it works
(I could certainly also store some data and read them back after realloc,
and also test one without omp_atk_alignment which effectively prevents
memkind_realloc from being called and uses allocation + deallocation), but
that is it.  I've actually stepped through in the debugger to verify
memkind_* is called...

Now for HBW memory, some googling around and brief look at the memkind
source shows that it probably supports just Intel Xeon Phi HBW memory,
I'm waiting for access to such a box right now but it might take a few days.

For the DAX stuff, I admit I don't know what it exactly is (what kind of hw
it needs).

> > --- libgomp/config/linux/allocator.c.jj       2022-06-08 08:58:23.197078191 +0200
> > +++ libgomp/config/linux/allocator.c  2022-06-08 09:39:15.108410730 +0200
> > @@ -0,0 +1,36 @@
> 
> > +#define _GNU_SOURCE
> > +#include "libgomp.h"
> > +#if defined(PLUGIN_SUPPORT) && defined(LIBGOMP_USE_PTHREADS)
> > +#define LIBGOMP_USE_MEMKIND
> > +#endif
> > +
> > +#include "../../../allocator.c"
> 
> Given this use of 'PLUGIN_SUPPORT' (and thus 'dlopen' etc.) for something
> different than libgomp plugins (offloading), might move 'DL_LIBS',
> 'PLUGIN_SUPPORT' from 'libgomp/plugin/configfrag.ac' into
> 'libgomp/configure.ac', and 'libgomp_la_LIBADD += $(DL_LIBS)' from
> 'libgomp/plugin/Makefrag.am' into 'libgomp/Makefile.am'.

Maybe, but libgomp/plugin/configfrag.ac is included unconditionally
and the memkind support is some kind of plugin too, just not offloading
plugin, but allocator plugin...
Didn't want to spend too much time on it and PLUGIN_SUPPORT
is right now solely about dlsym exists and -ldl works and has been added.

	Jakub