Re: [committed] openmp: Add support for HBW or large capacity or interleaved memory through the libmemkind.so library

[Andrew Stubbs <ams@codesourcery.com>]

On 09/06/2022 09:19, Jakub Jelinek via Gcc-patches wrote:
> +  switch (memspace)
> +    {
> +    case omp_high_bw_mem_space:
> +#ifdef LIBGOMP_USE_MEMKIND
> +      struct gomp_memkind_data *memkind_data;
> +      memkind_data = gomp_get_memkind ();
> +      if (data.partition == omp_atv_interleaved
> +	  && memkind_data->kinds[GOMP_MEMKIND_HBW_INTERLEAVE])
> +	{
> +	  data.memkind = GOMP_MEMKIND_HBW_INTERLEAVE;
> +	  break;
> +	}
> +      else if (memkind_data->kinds[GOMP_MEMKIND_HBW_PREFERRED])
> +	{
> +	  data.memkind = GOMP_MEMKIND_HBW_PREFERRED;
> +	  break;
> +	}
> +#endif
> +      return omp_null_allocator;
> +    case omp_large_cap_mem_space:
> +#ifdef LIBGOMP_USE_MEMKIND
> +      memkind_data = gomp_get_memkind ();
> +      if (memkind_data->kinds[GOMP_MEMKIND_DAX_KMEM_ALL])
> +	data.memkind = GOMP_MEMKIND_DAX_KMEM_ALL;
> +      else if (memkind_data->kinds[GOMP_MEMKIND_DAX_KMEM])
> +	data.memkind = GOMP_MEMKIND_DAX_KMEM;
> +#endif
> +      break;
> +    default:
> +#ifdef LIBGOMP_USE_MEMKIND
> +      if (data.partition == omp_atv_interleaved)
> +	{
> +	  memkind_data = gomp_get_memkind ();
> +	  if (memkind_data->kinds[GOMP_MEMKIND_INTERLEAVE])
> +	    data.memkind = GOMP_MEMKIND_INTERLEAVE;
> +	}
> +#endif
> +      break;
> +    }
> +

This conflicts with mine and Abid's patches to implement the device 
allocators and host unified shared memory via the overridable 
"MEMSPACE_ALLOC" hooks. I can still use those for the "else" case, but 
they'll be inactive on any configuration where libmemkind exists. That's 
fine for the device code, and may be OK for USM (given that libmemkind 
won't have an option for that). There's a problem for the 
NVidia-specific host-memory pinning I have planned though.

How do you propose we resolve this conflict, please?

Ideally it will be in such a way that the conditional is resolved at 
compile time and the routine can be inlined (so no fake-OO function 
pointers in structs, I think).

Thanks

Andrew