[Bug libgomp/113513] New: [OpenMP] libgomp: cuCtxGetDevice error with OMP_DISPLAY_ENV=true OMP_TARGET_OFFLOAD="mandatory" for libgomp.c/target-52.c

[Bug libgomp/113513] New: [OpenMP] libgomp: cuCtxGetDevice error with OMP_DISPLAY_ENV=true OMP_TARGET_OFFLOAD="mandatory" for libgomp.c/target-52.c

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113513

            Bug ID: 113513
           Summary: [OpenMP] libgomp: cuCtxGetDevice error with
                    OMP_DISPLAY_ENV=true OMP_TARGET_OFFLOAD="mandatory"
                    for libgomp.c/target-52.c
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Keywords: openmp, wrong-code
          Severity: normal
          Priority: P3
         Component: libgomp
          Assignee: unassigned at gcc dot gnu.org
          Reporter: burnus at gcc dot gnu.org
                CC: jakub at gcc dot gnu.org, tschwinge at gcc dot gnu.org
  Target Milestone: ---

When using both OMP_DISPLAY_ENV=true and OMP_TARGET_OFFLOAD="mandatory", the
device has to be initiated early as OMP_DEFAULT_DEVICE (either 0 or -4 =
omp_invalid_device) needs to be known before printing the ICVs.

On my system, this causes
  libgomp: cuCtxGetDevice error: unknown cuda error.

That's with "CUDA Version: 12.3" and "NVIDIA RTX A1000 6GB" with
--with-arch=sm_80.

I am somewhat sure that I have manually tested it before; our tester wasn't
able to remotely set the env vars, hence, I don't know whether it did work
there or not - nor whether it is a regression, depends on CUDA, sm_xx, my card
or ...

[Bug libgomp/113513] [OpenMP] libgomp: cuCtxGetDevice error with OMP_DISPLAY_ENV=true OMP_TARGET_OFFLOAD="mandatory" for libgomp.c/target-52.c

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113513

--- Comment #1 from Tobias Burnus <burnus at gcc dot gnu.org> ---
Looking at the called GOMP_OFFLOAD_* function, in the failing case, there is:

...
DEBUG GOMP_OFFLOAD_run
DEBUG GOMP_OFFLOAD_dev2host
DEBUG GOMP_OFFLOAD_free
DEBUG: nvptx_attach_host_thread_to_device - 0

and in the successful case:

DEBUG GOMP_OFFLOAD_fini_device 0  <<< called before unregister
DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_FINALIZED
DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_FINALIZED

and then - in the failing case:

DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_INITIALIZED
DEBUG: GOMP_offload_unregister_ver dev=0; state=GOMP_DEVICE_INITIALIZED
DEBUG: gomp_unload_image_from_device
DEBUG GOMP_OFFLOAD_unload_image, 0, 196609
DEBUG: gomp_target_fini; dev=0, state=GOMP_DEVICE_INITIALIZED
DEBUG GOMP_OFFLOAD_fini_device 0
DEBUG: nvptx_attach_host_thread_to_device - 0
libgomp: cuCtxGetDevice error: unknown cuda error


Thus, for some reason, GOMP_OFFLOAD_fini_device then
GOMP_offload_unregister_ver is swapped when
  OMP_DISPLAY_ENV=true and OMP_TARGET_OFFLOAD="mandatory"
are set - but not otherwise.


The call to omp_target_fini comes from:

  if (atexit (gomp_target_fini) != 0)
    gomp_fatal ("atexit failed");


While the call to   GOMP_offload_unregister_ver  comes from mkoffload:

  fprintf (out, "static __attribute__((destructor)) void fini (void)\n"
           "{\n"
           "  GOMP_offload_unregister_ver (%#x, __OFFLOAD_TABLE__,"
           " %d/*NVIDIA_PTX*/, &nvptx_data);\n"
           "};\n",


 * * * 


Actually, the same problem occurs when compiled with:

  -foffload=disable

With that flag + no 'mandatory':

DEBUG GOMP_OFFLOAD_version
DEBUG GOMP_OFFLOAD_get_caps
DEBUG GOMP_OFFLOAD_get_num_devices 0
DEBUG GOMP_OFFLOAD_get_name
DEBUG GOMP_OFFLOAD_get_type
DEBUG GOMP_OFFLOAD_init_device 0
DEBUG: nvptx_open_device - 0
DEBUG: gomp_target_fini; dev=0, state=GOMP_DEVICE_INITIALIZED
DEBUG GOMP_OFFLOAD_fini_device 0
DEBUG: nvptx_attach_host_thread_to_device - 0


And with 'mandatory' + OMP_DISPLAY_ENV=verbose:

DEBUG GOMP_OFFLOAD_version
DEBUG GOMP_OFFLOAD_get_caps
DEBUG GOMP_OFFLOAD_get_num_devices 0
DEBUG GOMP_OFFLOAD_get_name
DEBUG GOMP_OFFLOAD_get_type
< omp_display_env output>
DEBUG GOMP_OFFLOAD_init_device 0
DEBUG: nvptx_open_device - 0

libgomp: OMP_TARGET_OFFLOAD is set to MANDATORY, but device cannot be used for
offloading
DEBUG: gomp_target_fini; dev=0, state=GOMP_DEVICE_INITIALIZED
DEBUG GOMP_OFFLOAD_fini_device 0
DEBUG: nvptx_attach_host_thread_to_device - 

libgomp: cuCtxGetDevice error: unknown cuda error

libgomp: device finalization failed


Thus, the error message is the same – but here no offloading code exists and
just gomp_target_fini is called. - However, there is a prior call to 
'gomp_fatal'  which probably messes things up for the plugin handling - while
in the original code, we have a valid code.

 * * *

If there is no offloading code but
  OMP_DISPLAY_ENV=verbose OMP_TARGET_OFFLOAD="mandatory"
is used, it works:

DEBUG GOMP_OFFLOAD_version
DEBUG GOMP_OFFLOAD_get_caps
DEBUG GOMP_OFFLOAD_get_num_devices 0
DEBUG GOMP_OFFLOAD_get_name
DEBUG GOMP_OFFLOAD_get_type

OPENMP DISPLAY ENVIRONMENT BEGIN
 ...
OPENMP DISPLAY ENVIRONMENT END
DEBUG: gomp_target_fini; dev=0, state=0

 * * *

If there is only one or none of the two env vars, there is no need to search
for devices - and, hence, the nvptx plugin is not called at all and it,
obviously, works as well.

[Bug libgomp/113513] [OpenMP] libgomp: cuCtxGetDevice error with OMP_DISPLAY_ENV=true OMP_TARGET_OFFLOAD="mandatory" for libgomp.c/target-52.c

[burnus at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113513

--- Comment #2 from Tobias Burnus <burnus at gcc dot gnu.org> ---
Patch:
  https://gcc.gnu.org/pipermail/gcc-patches/2024-January/643648.html