[Bug libgomp/113513] [OpenMP] libgomp: cuCtxGetDevice error with OMP_DISPLAY_ENV=true OMP_TARGET_OFFLOAD="mandatory" for libgomp.c/target-52.c

["burnus at gcc dot gnu.org" <gcc-bugzilla@gcc.gnu.org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113513

--- Comment #1 from Tobias Burnus <burnus at gcc dot gnu.org> ---
Looking at the called GOMP_OFFLOAD_* function, in the failing case, there is:

...
DEBUG GOMP_OFFLOAD_run
DEBUG GOMP_OFFLOAD_dev2host
DEBUG GOMP_OFFLOAD_free
DEBUG: nvptx_attach_host_thread_to_device - 0

and in the successful case:

DEBUG GOMP_OFFLOAD_fini_device 0  <<< called before unregister
DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_FINALIZED
DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_FINALIZED

and then - in the failing case:

DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_INITIALIZED
DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_INITIALIZED
DEBUG: gomp_unload_image_from_device
DEBUG GOMP_OFFLOAD_unload_image, 0, 196609
DEBUG: gomp_target_fini; dev=0, state=GOMP_DEVICE_INITIALIZED
DEBUG GOMP_OFFLOAD_fini_device 0
DEBUG: nvptx_attach_host_thread_to_device - 0
libgomp: cuCtxGetDevice error: unknown cuda error


Thus, for some reason, GOMP_OFFLOAD_fini_device then
GOMP_offload_unregister_ver is swapped when
  OMP_DISPLAY_ENV=true and OMP_TARGET_OFFLOAD="mandatory"
are set - but not otherwise.


The call to omp_target_fini comes from:

  if (atexit (gomp_target_fini) != 0)
    gomp_fatal ("atexit failed");


While the call to   GOMP_offload_unregister_ver  comes from mkoffload:

  fprintf (out, "static __attribute__((destructor)) void fini (void)\n"
           "{\n"
           "  GOMP_offload_unregister_ver (%#x, __OFFLOAD_TABLE__,"
           " %d/*NVIDIA_PTX*/, &nvptx_data);\n"
           "};\n",


 * * * 


Actually, the same problem occurs when compiled with:

  -foffload=disable

With that flag + no 'mandatory':

DEBUG GOMP_OFFLOAD_version
DEBUG GOMP_OFFLOAD_get_caps
DEBUG GOMP_OFFLOAD_get_num_devices 0
DEBUG GOMP_OFFLOAD_get_name
DEBUG GOMP_OFFLOAD_get_type
DEBUG GOMP_OFFLOAD_init_device 0
DEBUG: nvptx_open_device - 0
DEBUG: gomp_target_fini; dev=0, state=GOMP_DEVICE_INITIALIZED
DEBUG GOMP_OFFLOAD_fini_device 0
DEBUG: nvptx_attach_host_thread_to_device - 0


And with 'mandatory' + OMP_DISPLAY_ENV=verbose:

DEBUG GOMP_OFFLOAD_version
DEBUG GOMP_OFFLOAD_get_caps
DEBUG GOMP_OFFLOAD_get_num_devices 0
DEBUG GOMP_OFFLOAD_get_name
DEBUG GOMP_OFFLOAD_get_type
< omp_display_env output>
DEBUG GOMP_OFFLOAD_init_device 0
DEBUG: nvptx_open_device - 0

libgomp: OMP_TARGET_OFFLOAD is set to MANDATORY, but device cannot be used for
offloading
DEBUG: gomp_target_fini; dev=0, state=GOMP_DEVICE_INITIALIZED
DEBUG GOMP_OFFLOAD_fini_device 0
DEBUG: nvptx_attach_host_thread_to_device - 

libgomp: cuCtxGetDevice error: unknown cuda error

libgomp: device finalization failed


Thus, the error message is the same – but here no offloading code exists and
just gomp_target_fini is called. - However, there is a prior call to 
'gomp_fatal'  which probably messes things up for the plugin handling - while
in the original code, we have a valid code.

 * * *

If there is no offloading code but
  OMP_DISPLAY_ENV=verbose OMP_TARGET_OFFLOAD="mandatory"
is used, it works:

DEBUG GOMP_OFFLOAD_version
DEBUG GOMP_OFFLOAD_get_caps
DEBUG GOMP_OFFLOAD_get_num_devices 0
DEBUG GOMP_OFFLOAD_get_name
DEBUG GOMP_OFFLOAD_get_type

OPENMP DISPLAY ENVIRONMENT BEGIN
 ...
OPENMP DISPLAY ENVIRONMENT END
DEBUG: gomp_target_fini; dev=0, state=0

 * * *

If there is only one or none of the two env vars, there is no need to search
for devices - and, hence, the nvptx plugin is not called at all and it,
obviously, works as well.