Re: [Patch] OpenMP: Prepare omp-* for ancestor:1 handling

[Tobias Burnus <tobias@codesourcery.com>]

Hi Jakub,

On 30.06.22 10:21, Jakub Jelinek wrote:
> So, what is the plan with reverse offload?

My idea was to just call omp_target_ext with
'device(omp_initial_device)'. This then automatically
works when called from a target region that runs on
omp_get_initial_device().

For the actual device part, this can be implemented
incrementally by supporting the reverse_offload for
a given device type.

For getting it to work when the code enclosing the ancestor:1
target region runs on an offloading device,
my idea is the following. Comments are welcome!


My idea was to do the same as done for I/O
(which supported for both nvptx and gcn). For GCN:

libgomp/plugin/plugin-gcn.c has:

struct kernargs {
   /* A pointer to struct output, below, for console output data.  */
   int64_t out_ptr;

   /* A pointer to struct heap, below.  */
   int64_t heap_ptr;

   /* A pointer to an ephemeral memory arena.
     Only needed for OpenMP.  */
   int64_t arena_ptr;

/* to be added: */
   /* A pointer to reverse-offload. */
   int64_t rev_ptr;

/* Now come the actual structs.*/
   /* Output data.  */
   struct output {
     int return_value;
     unsigned int next_output;
     struct printf_data {
...
};


This gets initialized on the host and then:

   while (hsa_fns.hsa_signal_wait_acquire_fn (s, HSA_SIGNAL_CONDITION_LT, 1,
                                              1000 * 1000,
                                              HSA_WAIT_STATE_BLOCKED) != 0)
     console_output (kernel, shadow->kernarg_address, false);

with:

   unsigned int from = __atomic_load_n (&kernargs->output_data.consumed,
                                        __ATOMIC_ACQUIRE);

The I/O itself is implemented in newlib,
https://sourceware.org/git/?p=newlib-cygwin.git;a=blob;f=newlib/libc/sys/amdgcn/write.c

   register void **kernargs asm("s8");
   struct output *data = (struct output *)kernargs[2];

and then the data is filled.


For reverse offload, the idea is fill it on the device side via
/libgomp/config/gcn/target.c's GOMP_target_ext for
device == GOMP_DEVICE_HOST_FALLBACK && fn != NULL as:

Try to obtain a lock (busy wait)
Put addr/kinds/sizes into the struct
Put the device's fn pointer in the struct
busy wait for completion ('while (fn != NULL) { }')
unlock


And on the host side:
If fn == NULL (= data there) - return output/offload checking loop
Otherwise:
call a new function in target.c and pass args to it.
Once it completed, set fn = NULL to indicate it has been processed.

And in target.c's new reverse-offload-handling function:
- find generated-target function on the host,
   based on device stub function's pointer address
- Handle the mapping
- Call host function
- Handle the mapping
- return

Additionally:

If 'requires reverse_offload' is set, fill not only
the normal splay_tree for "host -> device" lookup but
also another one for the "device -> host" lookups.

Does this make sense?

Tobias

-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955