Re: [v2][patch] plugin/plugin-nvptx.c: Fix fini_device call when already shutdown [PR113513]

[Tobias Burnus <tburnus@baylibre.com>]

[-- Attachment #1: Type: text/plain, Size: 6615 bytes --]

Hi Thomas,

Thomas Schwinge wrote:
> On 2024-01-23T10:55:16+0100, Tobias Burnus <tburnus@baylibre.com> wrote:
>> plugin/plugin-nvptx.c: Fix fini_device call when already shutdown [PR113513]
>>
>> The following issue was found when running libgomp.c/target-52.c with
>> nvptx offloading when the dg-set-target-env-var was honored.
> Curious, I've never seen this failure mode in my several different
> configurations.  :-|

I think we recently fixed a surprisingly high number of issues that we 
didn't see before but were clearly preexisting for quite a while. 
(Mostly for AMDGPU but still.)

But I concur that this one is a more tricky one.

>> For some unknown reasons, while this does not have an effect on the
>> order of the called plugin functions for initialization, it changes the
>> order of function calls for shutting down. Namely, when the two environment
>> variables are set, GOMP_offload_unregister_ver is called now before
>> gomp_target_fini.
> Re "unknown reasons", isn't that indeed explained by the different
> 'atexit' function/'__attribute__((destructor))' sequencing, due to
> different order of 'atexit'/'__attribute__((constructor))' calls?

Maybe or not. First, it does not seem to occur elsewhere but maybe 
that's because remote setting of environment variables does not work 
with DejaGNU and most code was run such a way. And secondly, I have no 
idea how 'atexit' and destructors are implemented internally.

>> And it seems as if CUDA regards a call to cuModuleUnload
>> (or unloading the last module?) as indication that the device context should
>> be destroyed - or, at least, afterwards calling cuCtxGetDevice will return
>> CUDA_ERROR_DEINITIALIZED.
> However, this I don't understand -- but would like to.  Are you saying
> that for:
>
>      --- libgomp/plugin/plugin-nvptx.c
>      +++ libgomp/plugin/plugin-nvptx.c
>      @@ -1556,8 +1556,16 @@ GOMP_OFFLOAD_unload_image (int ord, unsigned version, const void *target_data)
>           if (image->target_data == target_data)
>             {
>       	*prev_p = image->next;
>      -	if (CUDA_CALL_NOCHECK (cuModuleUnload, image->module) != CUDA_SUCCESS)
>      +	CUresult r;
>      +	r = CUDA_CALL_NOCHECK (cuModuleUnload, image->module);
>      +	GOMP_PLUGIN_debug (0, "%s: cuModuleUnload: %s\n", __FUNCTION__, cuda_error (r));
>      +	if (r != CUDA_SUCCESS)
>       	  ret = false;
>      +	CUdevice dev_;
>      +	r = CUDA_CALL_NOCHECK (cuCtxGetDevice, &dev_);
>      +	GOMP_PLUGIN_debug (0, "%s: cuCtxGetDevice: %s\n", __FUNCTION__, cuda_error (r));
>      +	GOMP_PLUGIN_debug (0, "%s: dev_=%d, dev->dev=%d\n", __FUNCTION__, dev_, dev->dev);
>      +	assert (dev_ == dev->dev);
>       	free (image->fns);
>       	free (image);
>       	break;
>
> ..., you're seeing an error for 'libgomp.c/target-52.c' with
> 'env OMP_TARGET_OFFLOAD=mandatory OMP_DISPLAY_ENV=true'?  I get:
>
>      GOMP_OFFLOAD_unload_image: cuModuleUnload: no error
>      GOMP_OFFLOAD_unload_image: cuCtxGetDevice: no error
>      GOMP_OFFLOAD_unload_image: dev_=0, dev->dev=0
>
> Or, is something else happening in between the 'cuModuleUnload' and your
> reportedly failing 'cuCtxGetDevice'?

I cluttered the plugin with "printf" debugging; hence, no other code
is calling *into* the run-time library as far as I can see.

But now I will try it with a vanilla code and your patch applied.

Result for the target-52.c with the env vars set:

DEBUG: GOMP_offload_unregister_ver dev=0; state=1
DEBUG: gomp_unload_image_from_device
DEBUG GOMP_OFFLOAD_unload_image, 0, 196609
GOMP_OFFLOAD_unload_image: cuModuleUnload: no error
GOMP_OFFLOAD_unload_image: cuCtxGetDevice: no error
GOMP_OFFLOAD_unload_image: dev_=0, dev->dev=0
DEBUG: gomp_target_fini; dev=0, state=1
DEBUG  0
DEBUG: nvptx_attach_host_thread_to_device - 0
DEBUG: ERROR nvptx_attach_host_thread_to_device - 0

libgomp: cuCtxGetDevice error: unknown cuda error

Hence: The immediately calling cuCtxGetDevice after
the device unloading does not fail.

But calling it soon late via gomp_target_fini
→ GOMP_OFFLOAD_fini_device → nvptx_attach_host_thread_to_device
does fail.

I have attached my printf patch for reference.

* * *

> Re your PR113513 details, I don't see how your failure mode could be
> related to (a) the PTX code ('--with-arch=sm_80'), or the GPU hardware
> ("NVIDIA RTX A1000 6GB") (..., unless the Nvidia Driver is doing "funny"
> things, of course...), so could this possibly be due to a recent change
> in the CUDA Driver/Nvidia Driver?  You say "CUDA Version: 12.3", but
> which which Nvidia Driver version?  The latest I've now tested are:
>
>      Driver Version: 525.147.05   CUDA Version: 12.0
>      Driver Version: 535.154.05   CUDA Version: 12.2

My laptop has:

NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3

> I'd like to please defer that one until we understand the actual origin
> of the misbehavior.
(I think that patch makes still sense, but first finding out what goes 
wrong is fine nonetheless.)
>> When reading the code, the following was observed in addition:
>> When gomp_fini_device is called, it invokes goacc_fini_asyncqueues
>> to ensure that the queue is emptied.  It seems to make sense to do
>> likewise for GOMP_offload_unregister_ver, which this commit does in
>> addition.
> I don't understand why offload image unregistration (a) should trigger
> 'goacc_fini_asyncqueues', and (b) how that relates to PR113513?

While there no direct relation and none to the testcase, this is 
affected by the ordering of GOMP_offload_unregister_ver vs.before 
gomp_target_fini, which is what the main issue is above.

Assume that by some reason GOMP_offload_unregister_ver gets called 
before gomp_target_fini. In that case, the asynchronous queues can be 
still running when the variables are removed and only when later 
gomp_target_fini is called, it will invoke goacc_fini_asyncqueues.

Of course, when gomp_target_fini is called first, it will run 
goacc_fini_asyncqueues first – and a later GOMP_offload_unregister_ver 
is a no op as the device is already finalized.

Thus, this part of the patch adds a safeguard for something to be a 
known issue for a related issue.

If we guarantee that gomp_target_fini is always called first, I suggest 
to remove GOMP_offload_unregister_ver for good as that will then be 
always unreachable ... (Well, that function itself not but it will not 
do any actual work.)

If we don't think so and there might be an ordering issue, I very much 
would like to see this safeguard in, which is very inexpensive if no 
work remains to be completed.

Tobias

[-- Attachment #2: debug_nvptx_fini.diff --]
[-- Type: text/x-patch, Size: 13310 bytes --]

diff --git a/libgomp/plugin/plugin-nvptx.c b/libgomp/plugin/plugin-nvptx.c
index c04c3acd679..7fc7f4a5bbf 100644
--- a/libgomp/plugin/plugin-nvptx.c
+++ b/libgomp/plugin/plugin-nvptx.c
@@ -391,7 +391,7 @@ nvptx_attach_host_thread_to_device (int n)
   CUresult r;
   struct ptx_device *ptx_dev;
   CUcontext thd_ctx;
-
+__builtin_fprintf (stderr, "DEBUG: nvptx_attach_host_thread_to_device - %d\n", n);
   r = CUDA_CALL_NOCHECK (cuCtxGetDevice, &dev);
   if (r == CUDA_ERROR_NOT_PERMITTED)
     {
@@ -400,6 +400,7 @@ nvptx_attach_host_thread_to_device (int n)
     }
   if (r != CUDA_SUCCESS && r != CUDA_ERROR_INVALID_CONTEXT)
     {
+__builtin_fprintf (stderr, "DEBUG: ERROR nvptx_attach_host_thread_to_device - %d\n", n);
       GOMP_PLUGIN_error ("cuCtxGetDevice error: %s", cuda_error (r));
       return false;
     }
@@ -445,9 +446,11 @@ nvptx_open_device (int n)
   ptx_dev->dev = dev;
   ptx_dev->ctx_shared = false;
 
+__builtin_fprintf (stderr, "DEBUG: nvptx_open_device - %d\n", n);
   r = CUDA_CALL_NOCHECK (cuCtxGetDevice, &ctx_dev);
   if (r != CUDA_SUCCESS && r != CUDA_ERROR_INVALID_CONTEXT)
     {
+__builtin_fprintf (stderr, "DEBUG: ERROR nvptx_open_device - %d\n", n);
       GOMP_PLUGIN_error ("cuCtxGetDevice error: %s", cuda_error (r));
       return NULL;
     }
@@ -1174,24 +1177,28 @@ nvptx_get_current_cuda_context (void)
 const char *
 GOMP_OFFLOAD_get_name (void)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_name\n");
   return "nvptx";
 }
 
 unsigned int
 GOMP_OFFLOAD_get_caps (void)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_caps\n");
   return GOMP_OFFLOAD_CAP_OPENACC_200 | GOMP_OFFLOAD_CAP_OPENMP_400;
 }
 
 int
 GOMP_OFFLOAD_get_type (void)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_type\n");
   return OFFLOAD_TARGET_TYPE_NVIDIA_PTX;
 }
 
 int
 GOMP_OFFLOAD_get_num_devices (unsigned int omp_requires_mask)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_num_devices %u\n", omp_requires_mask);
   int num_devices = nvptx_get_num_devices ();
   /* Return -1 if no omp_requires_mask cannot be fulfilled but
      devices were present.  Unified-shared address: see comment in
@@ -1207,6 +1214,7 @@ GOMP_OFFLOAD_get_num_devices (unsigned int omp_requires_mask)
 bool
 GOMP_OFFLOAD_init_device (int n)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_init_device %u\n", n);
   struct ptx_device *dev;
 
   pthread_mutex_lock (&ptx_dev_lock);
@@ -1248,6 +1256,7 @@ GOMP_OFFLOAD_init_device (int n)
 bool
 GOMP_OFFLOAD_fini_device (int n)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_fini_device %u\n", n);
   pthread_mutex_lock (&ptx_dev_lock);
 
   if (ptx_devices[n] != NULL)
@@ -1278,6 +1287,7 @@ GOMP_OFFLOAD_fini_device (int n)
 unsigned
 GOMP_OFFLOAD_version (void)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_version\n");
   return GOMP_VERSION;
 }
 
@@ -1311,6 +1321,7 @@ GOMP_OFFLOAD_load_image (int ord, unsigned version, const void *target_data,
 			 uint64_t **rev_fn_table,
 			 uint64_t *host_ind_fn_table)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_load_image, %d, %u\n", ord, version);
   CUmodule module;
   const char *const *var_names;
   const struct targ_fn_launch *fn_descs;
@@ -1538,6 +1549,7 @@ GOMP_OFFLOAD_load_image (int ord, unsigned version, const void *target_data,
 bool
 GOMP_OFFLOAD_unload_image (int ord, unsigned version, const void *target_data)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_unload_image, %d, %u\n", ord, version);
   struct ptx_image_data *image, **prev_p;
   struct ptx_device *dev = ptx_devices[ord];
 
@@ -1568,6 +1580,7 @@ GOMP_OFFLOAD_unload_image (int ord, unsigned version, const void *target_data)
 void *
 GOMP_OFFLOAD_alloc (int ord, size_t size)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_alloc, %d, %lu\n", ord, (long unsigned)size);
   if (!nvptx_attach_host_thread_to_device (ord))
     return NULL;
 
@@ -1604,6 +1617,7 @@ GOMP_OFFLOAD_alloc (int ord, size_t size)
 bool
 GOMP_OFFLOAD_free (int ord, void *ptr)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_free, %d\n", ord);
   return (nvptx_attach_host_thread_to_device (ord)
 	  && nvptx_free (ptr, ptx_devices[ord]));
 }
@@ -1615,6 +1629,7 @@ GOMP_OFFLOAD_openacc_exec (void (*fn) (void *),
 			   void **devaddrs,
 			   unsigned *dims, void *targ_mem_desc)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_exec\n");
   GOMP_PLUGIN_debug (0, "nvptx %s\n", __FUNCTION__);
 
   CUdeviceptr dp = (CUdeviceptr) devaddrs;
@@ -1637,6 +1652,7 @@ GOMP_OFFLOAD_openacc_async_exec (void (*fn) (void *),
 				 unsigned *dims, void *targ_mem_desc,
 				 struct goacc_asyncqueue *aq)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_exec\n");
   GOMP_PLUGIN_debug (0, "nvptx %s\n", __FUNCTION__);
 
   CUdeviceptr dp = (CUdeviceptr) devaddrs;
@@ -1646,6 +1662,7 @@ GOMP_OFFLOAD_openacc_async_exec (void (*fn) (void *),
 void *
 GOMP_OFFLOAD_openacc_create_thread_data (int ord)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_create_thread_data\n");
   struct ptx_device *ptx_dev;
   struct nvptx_thread *nvthd
     = GOMP_PLUGIN_malloc (sizeof (struct nvptx_thread));
@@ -1670,18 +1687,21 @@ GOMP_OFFLOAD_openacc_create_thread_data (int ord)
 void
 GOMP_OFFLOAD_openacc_destroy_thread_data (void *data)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_destroy_thread_data\n");
   free (data);
 }
 
 void *
 GOMP_OFFLOAD_openacc_cuda_get_current_device (void)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_get_current_device\n");
   return nvptx_get_current_cuda_device ();
 }
 
 void *
 GOMP_OFFLOAD_openacc_cuda_get_current_context (void)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_get_current_context\n");
   return nvptx_get_current_cuda_context ();
 }
 
@@ -1689,6 +1709,7 @@ GOMP_OFFLOAD_openacc_cuda_get_current_context (void)
 void *
 GOMP_OFFLOAD_openacc_cuda_get_stream (struct goacc_asyncqueue *aq)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_get_stream\n");
   return (void *) aq->cuda_stream;
 }
 
@@ -1696,6 +1717,7 @@ GOMP_OFFLOAD_openacc_cuda_get_stream (struct goacc_asyncqueue *aq)
 int
 GOMP_OFFLOAD_openacc_cuda_set_stream (struct goacc_asyncqueue *aq, void *stream)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_set_stream\n");
   if (aq->cuda_stream)
     {
       CUDA_CALL_ASSERT (cuStreamSynchronize, aq->cuda_stream);
@@ -1721,6 +1743,7 @@ nvptx_goacc_asyncqueue_construct (unsigned int flags)
 struct goacc_asyncqueue *
 GOMP_OFFLOAD_openacc_async_construct (int device __attribute__((unused)))
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_construct\n");
   return nvptx_goacc_asyncqueue_construct (CU_STREAM_DEFAULT);
 }
 
@@ -1735,12 +1758,14 @@ nvptx_goacc_asyncqueue_destruct (struct goacc_asyncqueue *aq)
 bool
 GOMP_OFFLOAD_openacc_async_destruct (struct goacc_asyncqueue *aq)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_destruct\n");
   return nvptx_goacc_asyncqueue_destruct (aq);
 }
 
 int
 GOMP_OFFLOAD_openacc_async_test (struct goacc_asyncqueue *aq)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_test\n");
   CUresult r = CUDA_CALL_NOCHECK (cuStreamQuery, aq->cuda_stream);
   if (r == CUDA_SUCCESS)
     return 1;
@@ -1761,6 +1786,7 @@ nvptx_goacc_asyncqueue_synchronize (struct goacc_asyncqueue *aq)
 bool
 GOMP_OFFLOAD_openacc_async_synchronize (struct goacc_asyncqueue *aq)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_synchronize\n");
   return nvptx_goacc_asyncqueue_synchronize (aq);
 }
 
@@ -1768,6 +1794,7 @@ bool
 GOMP_OFFLOAD_openacc_async_serialize (struct goacc_asyncqueue *aq1,
 				      struct goacc_asyncqueue *aq2)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_serialize\n");
   CUevent e;
   CUDA_CALL_ERET (false, cuEventCreate, &e, CU_EVENT_DISABLE_TIMING);
   CUDA_CALL_ERET (false, cuEventRecord, e, aq1->cuda_stream);
@@ -1790,6 +1817,7 @@ GOMP_OFFLOAD_openacc_async_queue_callback (struct goacc_asyncqueue *aq,
 					   void (*callback_fn)(void *),
 					   void *userptr)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_queue_callback\n");
   struct nvptx_callback *b = GOMP_PLUGIN_malloc (sizeof (*b));
   b->fn = callback_fn;
   b->ptr = userptr;
@@ -1837,6 +1865,7 @@ cuda_memcpy_sanity_check (const void *h, const void *d, size_t s)
 bool
 GOMP_OFFLOAD_host2dev (int ord, void *dst, const void *src, size_t n)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_host2dev\n");
   if (!nvptx_attach_host_thread_to_device (ord)
       || !cuda_memcpy_sanity_check (src, dst, n))
     return false;
@@ -1847,6 +1876,7 @@ GOMP_OFFLOAD_host2dev (int ord, void *dst, const void *src, size_t n)
 bool
 GOMP_OFFLOAD_dev2host (int ord, void *dst, const void *src, size_t n)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_dev2host\n");
   if (!nvptx_attach_host_thread_to_device (ord)
       || !cuda_memcpy_sanity_check (dst, src, n))
     return false;
@@ -1857,6 +1887,7 @@ GOMP_OFFLOAD_dev2host (int ord, void *dst, const void *src, size_t n)
 bool
 GOMP_OFFLOAD_dev2dev (int ord, void *dst, const void *src, size_t n)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_dev2dev\n");
   CUDA_CALL (cuMemcpyDtoDAsync, (CUdeviceptr) dst, (CUdeviceptr) src, n, NULL);
   return true;
 }
@@ -1868,6 +1899,7 @@ GOMP_OFFLOAD_memcpy2d (int dst_ord, int src_ord, size_t dim1_size,
 		       const void *src, size_t src_offset1_size,
 		       size_t src_offset0_len, size_t src_dim1_size)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_memcpy2d\n");
   if (!nvptx_attach_host_thread_to_device (src_ord != -1 ? src_ord : dst_ord))
     return false;
 
@@ -1960,6 +1992,7 @@ GOMP_OFFLOAD_memcpy3d (int dst_ord, int src_ord, size_t dim2_size,
 		       size_t src_offset0_len, size_t src_dim2_size,
 		       size_t src_dim1_len)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_memcpy3d\n");
   if (!nvptx_attach_host_thread_to_device (src_ord != -1 ? src_ord : dst_ord))
     return false;
 
@@ -2050,6 +2083,7 @@ bool
 GOMP_OFFLOAD_openacc_async_host2dev (int ord, void *dst, const void *src,
 				     size_t n, struct goacc_asyncqueue *aq)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_host2dev\n");
   if (!nvptx_attach_host_thread_to_device (ord)
       || !cuda_memcpy_sanity_check (src, dst, n))
     return false;
@@ -2061,6 +2095,7 @@ bool
 GOMP_OFFLOAD_openacc_async_dev2host (int ord, void *dst, const void *src,
 				     size_t n, struct goacc_asyncqueue *aq)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_dev2host\n");
   if (!nvptx_attach_host_thread_to_device (ord)
       || !cuda_memcpy_sanity_check (dst, src, n))
     return false;
@@ -2071,6 +2106,7 @@ GOMP_OFFLOAD_openacc_async_dev2host (int ord, void *dst, const void *src,
 union goacc_property_value
 GOMP_OFFLOAD_openacc_get_property (int n, enum goacc_property prop)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_get_property\n");
   union goacc_property_value propval = { .val = 0 };
 
   pthread_mutex_lock (&ptx_dev_lock);
@@ -2211,6 +2247,7 @@ nvptx_stacks_acquire (struct ptx_device *ptx_dev, size_t size, int num)
 void
 GOMP_OFFLOAD_run (int ord, void *tgt_fn, void *tgt_vars, void **args)
 {
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_run\n");
   struct targ_fn_descriptor *tgt_fn_desc
     = (struct targ_fn_descriptor *) tgt_fn;
   CUfunction function = tgt_fn_desc->fn;
diff --git a/libgomp/target.c b/libgomp/target.c
index 1367e9cce6c..f758b20ba4c 100644
--- a/libgomp/target.c
+++ b/libgomp/target.c
@@ -2524,6 +2524,8 @@ gomp_unload_image_from_device (struct gomp_device_descr *devicep,
       node = splay_tree_lookup (&devicep->mem_map, &k);
     }
 
+__builtin_fprintf(stderr, "DEBUG: gomp_unload_image_from_device\n");
+
   if (!devicep->unload_image_func (devicep->target_id, version, target_data))
     {
       gomp_mutex_unlock (&devicep->lock);
@@ -2698,12 +2700,14 @@ GOMP_offload_unregister_ver (unsigned version, const void *host_table,
     target_data = data;
 
   gomp_mutex_lock (&register_lock);
+__builtin_fprintf(stderr, "DEBUG: GOMP_offload_unregister_ver\n");
 
   /* Unload image from all initialized devices.  */
   for (i = 0; i < num_devices; i++)
     {
       struct gomp_device_descr *devicep = &devices[i];
       gomp_mutex_lock (&devicep->lock);
+__builtin_fprintf(stderr, "DEBUG: GOMP_offload_unregister_ver dev=%d; state=%d\n", i, devicep->state);
       if (devicep->type == target_type
 	  && devicep->state == GOMP_DEVICE_INITIALIZED)
 	gomp_unload_image_from_device (devicep, version,
@@ -2775,6 +2779,7 @@ gomp_fini_device (struct gomp_device_descr *devicep)
 attribute_hidden void
 gomp_unload_device (struct gomp_device_descr *devicep)
 {
+__builtin_fprintf(stderr, "DEBUG: gomp_unload_device; state=%d\n", devicep->state);
   if (devicep->state == GOMP_DEVICE_INITIALIZED)
     {
       unsigned i;
@@ -5217,6 +5222,7 @@ gomp_target_fini (void)
       bool ret = true;
       struct gomp_device_descr *devicep = &devices[i];
       gomp_mutex_lock (&devicep->lock);
+__builtin_fprintf(stderr, "DEBUG: gomp_target_fini; dev=%d, state=%d\n", i, devicep->state);
       if (devicep->state == GOMP_DEVICE_INITIALIZED)
 	ret = gomp_fini_device (devicep);
       gomp_mutex_unlock (&devicep->lock);