Re: libgomp nvptx plugin: Split 'PLUGIN_NVPTX_DYNAMIC' into 'PLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H' and 'PLUGIN_NVPTX_LINK_LIBCUDA'

[Thomas Schwinge <thomas@codesourcery.com>]

[-- Attachment #1: Type: text/plain, Size: 2878 bytes --]

Hi Tom!

On 2022-04-08T09:35:44+0200, Tom de Vries <tdevries@suse.de> wrote:
> On 4/8/22 00:27, Thomas Schwinge wrote:
>> On 2017-01-13T19:11:23+0100, Jakub Jelinek <jakub@redhat.com> wrote:
>>> Especially for distributions it is undesirable to need to have proprietary
>>> CUDA libraries and headers installed when building GCC.
>>
>>> --- libgomp/plugin/configfrag.ac.jj   2017-01-13 12:07:56.000000000 +0100
>>> +++ libgomp/plugin/configfrag.ac      2017-01-13 17:33:26.608240936 +0100
>>
>>> +           PLUGIN_NVPTX_CPPFLAGS='-I$(srcdir)/plugin/cuda'
>>> +           PLUGIN_NVPTX_LIBS='-ldl'
>>> +           PLUGIN_NVPTX_DYNAMIC=1
>>
>>> +AC_DEFINE_UNQUOTED([PLUGIN_NVPTX_DYNAMIC], [$PLUGIN_NVPTX_DYNAMIC],
>>> +  [Define to 1 if the NVIDIA plugin should dlopen libcuda.so.1, 0 if it should be linked against it.])
>>
>> Actually, the conditionals leading to 'PLUGIN_NVPTX_DYNAMIC=1' here do
>> control two orthogonal aspects; OK to disentangle that with the attached
>> "libgomp nvptx plugin: Split 'PLUGIN_NVPTX_DYNAMIC' into
>> 'PLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H' and 'PLUGIN_NVPTX_LINK_LIBCUDA'"?

> we discussed dropping --with-cuda, so do I understand it correctly that
> you now propose to drop --with-cuda and --with-cuda-driver-lib but
> intend to keep --with-cuda-driver-include ?

No, I think you're reading too much into this first patch.  ;-)

The goal with this patch is just to help disentangle two orthogonal
concepts (as described in the commit log), and then...

> Can you explain what user or maintainer scenario is served by this?

... in a next step, we may indeed remove the current user-visible
'--with-cuda-driver' etc., but keep the underlying functionality
available for the developers.  That's to address the point you'd made in
the "Proposal to remove '--with-cuda-driver'" thread: that it still
"could be useful for debugging / comparison purposes" -- and especially
for development purposes, in my opinion: if you develop CUDA API-level
changes in the libgomp nvptx plugin, it's likely to be easier to just use
the full CUDA toolkit 'cuda.h' and directly link against libcuda (so that
you've got all symbols etc. available), and only once you know what
exactly you need, update GCC's 'include/cuda/cuda.h' and
'libgomp/plugin/cuda-lib.def'.

With that hopefully clarified, OK to push the re-attached
"libgomp nvptx plugin: Split 'PLUGIN_NVPTX_DYNAMIC' into
'PLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H' and 'PLUGIN_NVPTX_LINK_LIBCUDA'"?

> Is
> there a problem with using gcc's cuda.h?

No, all good.


Grüße
 Thomas


-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-libgomp-nvptx-plugin-Split-PLUGIN_NVPTX_DYNAMIC-into.patch --]
[-- Type: text/x-diff, Size: 9643 bytes --]

From c455522ac5d8ab41e5d11f8997678e042ff48e87 Mon Sep 17 00:00:00 2001
From: Thomas Schwinge <thomas@codesourcery.com>
Date: Thu, 7 Apr 2022 23:10:16 +0200
Subject: [PATCH] libgomp nvptx plugin: Split 'PLUGIN_NVPTX_DYNAMIC' into
 'PLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H' and 'PLUGIN_NVPTX_LINK_LIBCUDA'

Including the GCC-shipped 'include/cuda/cuda.h' vs. system <cuda.h> and
'dlopen'ing the CUDA Driver library vs. linking it are separate concerns.

	libgomp/
	* plugin/Makefrag.am: Handle 'PLUGIN_NVPTX_DYNAMIC'.
	* plugin/configfrag.ac (PLUGIN_NVPTX_DYNAMIC): Change
	'AC_DEFINE_UNQUOTED' into 'AM_CONDITIONAL'.
	* plugin/plugin-nvptx.c: Split 'PLUGIN_NVPTX_DYNAMIC' into
	'PLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H' and
	'PLUGIN_NVPTX_LINK_LIBCUDA'.
	* Makefile.in: Regenerate.
	* config.h.in: Likewise.
	* configure: Likewise.
---
 libgomp/Makefile.in           | 26 +++++++++++++++++++-------
 libgomp/config.h.in           |  4 ----
 libgomp/configure             | 21 +++++++++++++++------
 libgomp/plugin/Makefrag.am    | 16 +++++++++++++++-
 libgomp/plugin/configfrag.ac  |  3 +--
 libgomp/plugin/plugin-nvptx.c |  4 ++--
 6 files changed, 52 insertions(+), 22 deletions(-)

diff --git a/libgomp/Makefile.in b/libgomp/Makefile.in
index 22cb2136a08..d43c584a32d 100644
--- a/libgomp/Makefile.in
+++ b/libgomp/Makefile.in
@@ -119,8 +119,16 @@ build_triplet = @build@
 host_triplet = @host@
 target_triplet = @target@
 @PLUGIN_NVPTX_TRUE@am__append_1 = libgomp-plugin-nvptx.la
-@PLUGIN_GCN_TRUE@am__append_2 = libgomp-plugin-gcn.la
-@USE_FORTRAN_TRUE@am__append_3 = openacc.f90
+
+# Including the GCC-shipped 'include/cuda/cuda.h' vs. system <cuda.h>.
+@PLUGIN_NVPTX_DYNAMIC_FALSE@@PLUGIN_NVPTX_TRUE@am__append_2 = -DPLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H \
+@PLUGIN_NVPTX_DYNAMIC_FALSE@@PLUGIN_NVPTX_TRUE@	-DPLUGIN_NVPTX_LINK_LIBCUDA
+
+# 'dlopen'ing the CUDA Driver library vs. linking it.
+@PLUGIN_NVPTX_DYNAMIC_TRUE@@PLUGIN_NVPTX_TRUE@am__append_3 = $(PLUGIN_NVPTX_LIBS)
+@PLUGIN_NVPTX_DYNAMIC_FALSE@@PLUGIN_NVPTX_TRUE@am__append_4 = $(PLUGIN_NVPTX_LIBS)
+@PLUGIN_GCN_TRUE@am__append_5 = libgomp-plugin-gcn.la
+@USE_FORTRAN_TRUE@am__append_6 = openacc.f90
 subdir = .
 ACLOCAL_M4 = $(top_srcdir)/aclocal.m4
 am__aclocal_m4_deps = $(top_srcdir)/../config/acx.m4 \
@@ -197,8 +205,10 @@ libgomp_plugin_gcn_la_LINK = $(LIBTOOL) $(AM_V_lt) --tag=CC \
 	$(libgomp_plugin_gcn_la_LDFLAGS) $(LDFLAGS) -o $@
 @PLUGIN_GCN_TRUE@am_libgomp_plugin_gcn_la_rpath = -rpath \
 @PLUGIN_GCN_TRUE@	$(toolexeclibdir)
+@PLUGIN_NVPTX_DYNAMIC_TRUE@@PLUGIN_NVPTX_TRUE@am__DEPENDENCIES_2 = $(am__DEPENDENCIES_1)
+@PLUGIN_NVPTX_DYNAMIC_FALSE@@PLUGIN_NVPTX_TRUE@am__DEPENDENCIES_3 = $(am__DEPENDENCIES_1)
 @PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_DEPENDENCIES = libgomp.la \
-@PLUGIN_NVPTX_TRUE@	$(am__DEPENDENCIES_1)
+@PLUGIN_NVPTX_TRUE@	$(am__DEPENDENCIES_2) $(am__DEPENDENCIES_3)
 @PLUGIN_NVPTX_TRUE@am_libgomp_plugin_nvptx_la_OBJECTS =  \
 @PLUGIN_NVPTX_TRUE@	libgomp_plugin_nvptx_la-plugin-nvptx.lo
 libgomp_plugin_nvptx_la_OBJECTS =  \
@@ -533,7 +543,7 @@ libsubincludedir = $(libdir)/gcc/$(target_alias)/$(gcc_version)/include
 AM_CPPFLAGS = $(addprefix -I, $(search_path))
 AM_CFLAGS = $(XCFLAGS)
 AM_LDFLAGS = $(XLDFLAGS) $(SECTION_LDFLAGS) $(OPT_LDFLAGS)
-toolexeclib_LTLIBRARIES = libgomp.la $(am__append_1) $(am__append_2)
+toolexeclib_LTLIBRARIES = libgomp.la $(am__append_1) $(am__append_5)
 nodist_toolexeclib_HEADERS = libgomp.spec
 
 # -Wc is only a libtool option.
@@ -559,16 +569,18 @@ libgomp_la_SOURCES = alloc.c atomic.c barrier.c critical.c env.c \
 	oacc-parallel.c oacc-host.c oacc-init.c oacc-mem.c \
 	oacc-async.c oacc-plugin.c oacc-cuda.c priority_queue.c \
 	affinity-fmt.c teams.c allocator.c oacc-profiling.c \
-	oacc-target.c $(am__append_3)
+	oacc-target.c $(am__append_6)
 
 # Nvidia PTX OpenACC plugin.
 @PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_version_info = -version-info $(libtool_VERSION)
 @PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_SOURCES = plugin/plugin-nvptx.c
-@PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_CPPFLAGS = $(AM_CPPFLAGS) $(PLUGIN_NVPTX_CPPFLAGS)
+@PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_CPPFLAGS = $(AM_CPPFLAGS) \
+@PLUGIN_NVPTX_TRUE@	$(PLUGIN_NVPTX_CPPFLAGS) $(am__append_2)
 @PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_LDFLAGS =  \
 @PLUGIN_NVPTX_TRUE@	$(libgomp_plugin_nvptx_version_info) \
 @PLUGIN_NVPTX_TRUE@	$(lt_host_flags) $(PLUGIN_NVPTX_LDFLAGS)
-@PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_LIBADD = libgomp.la $(PLUGIN_NVPTX_LIBS)
+@PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_LIBADD = libgomp.la \
+@PLUGIN_NVPTX_TRUE@	$(am__append_3) $(am__append_4)
 @PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_la_LIBTOOLFLAGS = --tag=disable-static
 
 # AMD GCN plugin
diff --git a/libgomp/config.h.in b/libgomp/config.h.in
index e702625ab6e..8d703ec7226 100644
--- a/libgomp/config.h.in
+++ b/libgomp/config.h.in
@@ -176,10 +176,6 @@
 /* Define to 1 if the NVIDIA plugin is built, 0 if not. */
 #undef PLUGIN_NVPTX
 
-/* Define to 1 if the NVIDIA plugin should dlopen libcuda.so.1, 0 if it should
-   be linked against it. */
-#undef PLUGIN_NVPTX_DYNAMIC
-
 /* Define if all infrastructure, needed for plugins, is supported. */
 #undef PLUGIN_SUPPORT
 
diff --git a/libgomp/configure b/libgomp/configure
index 5ef071ea046..447d1734930 100755
--- a/libgomp/configure
+++ b/libgomp/configure
@@ -667,6 +667,8 @@ OPT_LDFLAGS
 SECTION_LDFLAGS
 PLUGIN_GCN_FALSE
 PLUGIN_GCN_TRUE
+PLUGIN_NVPTX_DYNAMIC_FALSE
+PLUGIN_NVPTX_DYNAMIC_TRUE
 PLUGIN_NVPTX_FALSE
 PLUGIN_NVPTX_TRUE
 offload_additional_lib_paths
@@ -11431,7 +11433,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 11434 "configure"
+#line 11436 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -11537,7 +11539,7 @@ else
   lt_dlunknown=0; lt_dlno_uscore=1; lt_dlneed_uscore=2
   lt_status=$lt_dlunknown
   cat > conftest.$ac_ext <<_LT_EOF
-#line 11540 "configure"
+#line 11542 "configure"
 #include "confdefs.h"
 
 #if HAVE_DLFCN_H
@@ -15444,10 +15446,13 @@ cat >>confdefs.h <<_ACEOF
 #define PLUGIN_NVPTX $PLUGIN_NVPTX
 _ACEOF
 
-
-cat >>confdefs.h <<_ACEOF
-#define PLUGIN_NVPTX_DYNAMIC $PLUGIN_NVPTX_DYNAMIC
-_ACEOF
+ if test $PLUGIN_NVPTX_DYNAMIC = 1; then
+  PLUGIN_NVPTX_DYNAMIC_TRUE=
+  PLUGIN_NVPTX_DYNAMIC_FALSE='#'
+else
+  PLUGIN_NVPTX_DYNAMIC_TRUE='#'
+  PLUGIN_NVPTX_DYNAMIC_FALSE=
+fi
 
  if test $PLUGIN_GCN = 1; then
   PLUGIN_GCN_TRUE=
@@ -17221,6 +17226,10 @@ if test -z "${PLUGIN_NVPTX_TRUE}" && test -z "${PLUGIN_NVPTX_FALSE}"; then
   as_fn_error $? "conditional \"PLUGIN_NVPTX\" was never defined.
 Usually this means the macro was only invoked conditionally." "$LINENO" 5
 fi
+if test -z "${PLUGIN_NVPTX_DYNAMIC_TRUE}" && test -z "${PLUGIN_NVPTX_DYNAMIC_FALSE}"; then
+  as_fn_error $? "conditional \"PLUGIN_NVPTX_DYNAMIC\" was never defined.
+Usually this means the macro was only invoked conditionally." "$LINENO" 5
+fi
 if test -z "${PLUGIN_GCN_TRUE}" && test -z "${PLUGIN_GCN_FALSE}"; then
   as_fn_error $? "conditional \"PLUGIN_GCN\" was never defined.
 Usually this means the macro was only invoked conditionally." "$LINENO" 5
diff --git a/libgomp/plugin/Makefrag.am b/libgomp/plugin/Makefrag.am
index 3fe50b61cfd..d33df85b83e 100644
--- a/libgomp/plugin/Makefrag.am
+++ b/libgomp/plugin/Makefrag.am
@@ -35,8 +35,22 @@ libgomp_plugin_nvptx_la_CPPFLAGS = $(AM_CPPFLAGS) $(PLUGIN_NVPTX_CPPFLAGS)
 libgomp_plugin_nvptx_la_LDFLAGS = $(libgomp_plugin_nvptx_version_info) \
 	$(lt_host_flags)
 libgomp_plugin_nvptx_la_LDFLAGS += $(PLUGIN_NVPTX_LDFLAGS)
-libgomp_plugin_nvptx_la_LIBADD = libgomp.la $(PLUGIN_NVPTX_LIBS)
+libgomp_plugin_nvptx_la_LIBADD = libgomp.la
 libgomp_plugin_nvptx_la_LIBTOOLFLAGS = --tag=disable-static
+
+# Including the GCC-shipped 'include/cuda/cuda.h' vs. system <cuda.h>.
+if PLUGIN_NVPTX_DYNAMIC
+else
+libgomp_plugin_nvptx_la_CPPFLAGS += -DPLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H
+endif
+
+# 'dlopen'ing the CUDA Driver library vs. linking it.
+if PLUGIN_NVPTX_DYNAMIC
+libgomp_plugin_nvptx_la_LIBADD += $(PLUGIN_NVPTX_LIBS)
+else
+libgomp_plugin_nvptx_la_CPPFLAGS += -DPLUGIN_NVPTX_LINK_LIBCUDA
+libgomp_plugin_nvptx_la_LIBADD += $(PLUGIN_NVPTX_LIBS)
+endif
 endif
 
 if PLUGIN_GCN
diff --git a/libgomp/plugin/configfrag.ac b/libgomp/plugin/configfrag.ac
index 09f87f48bc7..6f5a1ba2b51 100644
--- a/libgomp/plugin/configfrag.ac
+++ b/libgomp/plugin/configfrag.ac
@@ -266,8 +266,7 @@ AC_DEFINE_UNQUOTED(OFFLOAD_PLUGINS, "$offload_plugins",
 AM_CONDITIONAL([PLUGIN_NVPTX], [test $PLUGIN_NVPTX = 1])
 AC_DEFINE_UNQUOTED([PLUGIN_NVPTX], [$PLUGIN_NVPTX],
   [Define to 1 if the NVIDIA plugin is built, 0 if not.])
-AC_DEFINE_UNQUOTED([PLUGIN_NVPTX_DYNAMIC], [$PLUGIN_NVPTX_DYNAMIC],
-  [Define to 1 if the NVIDIA plugin should dlopen libcuda.so.1, 0 if it should be linked against it.])
+AM_CONDITIONAL([PLUGIN_NVPTX_DYNAMIC], [test $PLUGIN_NVPTX_DYNAMIC = 1])
 AM_CONDITIONAL([PLUGIN_GCN], [test $PLUGIN_GCN = 1])
 AC_DEFINE_UNQUOTED([PLUGIN_GCN], [$PLUGIN_GCN],
   [Define to 1 if the GCN plugin is built, 0 if not.])
diff --git a/libgomp/plugin/plugin-nvptx.c b/libgomp/plugin/plugin-nvptx.c
index b28dfca00fa..387bcbbc52a 100644
--- a/libgomp/plugin/plugin-nvptx.c
+++ b/libgomp/plugin/plugin-nvptx.c
@@ -41,7 +41,7 @@
 #include "oacc-int.h"
 
 #include <pthread.h>
-#if PLUGIN_NVPTX_DYNAMIC
+#ifndef PLUGIN_NVPTX_INCLUDE_SYSTEM_CUDA_H
 # include "cuda/cuda.h"
 #else
 # include <cuda.h>
@@ -85,7 +85,7 @@ CUresult cuOccupancyMaxPotentialBlockSize(int *, int *, CUfunction,
 
 #define DO_PRAGMA(x) _Pragma (#x)
 
-#if PLUGIN_NVPTX_DYNAMIC
+#ifndef PLUGIN_NVPTX_LINK_LIBCUDA
 # include <dlfcn.h>
 
 struct cuda_lib_s {
-- 
2.25.1