Re: OPMD Task_Handles

[Jakub Jelinek <jakub@redhat.com>]

On Tue, May 31, 2022 at 02:01:34PM +0200, Yousssef Magdy wrote:
> hello there,
> I am currently implementing the ompd lib as a graduation project with
> Mohamed Atef, I am Facing some problems such as:
> - in  ompd_get_scheduling_task_handle
> <https://www.openmp.org/spec-html/5.1/openmpsu246.html#x311-4730005.5.7.3> that
> is supposed to get the task was active when the task that represents
> task_handle was scheduled, which is still on the stack of execution on the
> same thread. I see that there is some scheduling point, such that barriers
> and taskwaits.
> so is there any hint of how can i get the scheduling_task?

We implement untied tasks as tied, so I believe we never have threads that
would be deferred in favor of executing the current task.
So always ompd_rc_unavailable (after performing some basic checks)?

> - the other function ompd_get_task_frame
> <https://www.openmp.org/spec-html/5.1/openmpsu246.html#x311-4780005.5.7.8>
> should get the task exit_frame and enter_frame that point to the address of
>  ompt_frame_t
> <https://www.openmp.org/spec-html/5.1/openmpsu212.html#x269-3350004.4.4.28>
> which
> is dependent on ompt lib,  how could I implement it?

This means I'm afraid (not very happy about that) that we'll need to add
2 pointers to struct gomp_task and fill them up with
__builtin_frame_address(0) at certain spots, ideally guarded with
 if (__builtin_expect (gompd_enabled, 0))
   {
     // whatever, e.g. including
     // task->exit_frame = __builtin_frame_address (0);
   }
From my skimming of the spec (wasn't aware of such stuff until now at all),
exit_frame should be the outermost frame pointer that executes the task
region, and enter_frame NULL unless the current thread now executes
something else and that task will resume later on.
So if we have say
#include <unistd.h>

int
main ()
{
  #pragma omp parallel
  {
    #pragma omp single
    #pragma omp task
    {
      #pragma omp task if (0)
      {
	sleep (5);
      }
      #pragma omp parallel
      {
	sleep (10);
      }
    }
  }
}

one puts a breakpoint on the first sleep, the backtrace can be like:
#0  main._omp_fn.2 () at jjjj.c:13
#1  0x00007ffff7f8ed0b in GOMP_task (fn=<optimized out>, data=0x0, cpyfn=<optimized out>, arg_size=<optimized out>, arg_align=<optimized out>, if_clause=<optimized out>, 
    flags=<optimized out>, depend=<optimized out>, priority_arg=<optimized out>, detach=<optimized out>) at ../../../libgomp/task.c:584
#2  0x00000000004011fc in main._omp_fn.1 () at jjjj.c:9
#3  0x00007ffff7f8cbf9 in gomp_barrier_handle_tasks (state=<optimized out>) at ../../../libgomp/task.c:1650
#4  0x00007ffff7f95a20 in gomp_team_barrier_wait_end (bar=0x405440, state=0) at ../../../libgomp/config/linux/bar.c:116
#5  0x00007ffff7f95b0e in gomp_team_barrier_wait_final (bar=bar@entry=0x405440) at ../../../libgomp/config/linux/bar.c:136
#6  0x00007ffff7f92efa in gomp_thread_start (xdata=<optimized out>) at ../../../libgomp/team.c:130
#7  0x00007ffff7dd6e5d in start_thread () from /lib64/libc.so.6
#8  0x00007ffff7e5c8f0 in clone3 () from /lib64/libc.so.6

main._omp_fn.2 is the outlined function that contains { sleep (5); },
GOMP_task is the libgomp routine that starts an explicit task and in case of
an included task like if (0) or when for whatever reason that isnt deferred
calls the outlined function.
main._omp_fn.1 is the outlined function that has #pragma omp task if (0) and
#pragma omp parallel, and because that task isn't included, it isn't invoked
from GOMP_task, but from various other places like
gomp_barrier_handle_tasks, GOMP_taskwait,
gomp_task_maybe_wait_for_dependencies, GOMP_taskgroup_end.

Now, there are 2-3 different OpenMP tasks related to the current thread
- the implicit task created for the parallel, but that one has exited its
  user body main._omp_fn.0 already, is executing the implicit barrier
  at the end of the parallel, not sure if OMPD should see that still as
  active or not
- the outer explicit task
- the inner explicit task (with if (0))

For the last one, enter_frame should have NULL ptr, exit_frame can be
either the main._omp_fn.2 frame (then it would be ompt_frame_application)
or the GOMP_task frame (then it would be ompt_frame_runtime).  As it would
be really hard to figure out frame pointer of the user code, I'd strongly
suggest to use __builtin_frame_address (0) of the GOMP_task in that case.

The middle one, enter_frame should be probably the GOMP_task frame too,
and exit_frame again either main._omp_fn.1 frame or better
gomp_barrier_handle_tasks frame.

And the last one if we still pretend the implicit task is active would
have enter_frame of gomp_thread_start or so and exit_frame of
gomp_barrier_handle_tasks.  Other possibility is to say that it no longer
has one or more frames on the stack of a native thread and so doesn't have
corresponding frame object.

Similarly for case where breakpoint is on sleep (10) etc.

Now, if all the frame addresses we remember for something are in the libgomp
library or NULL, perhaps we can just use ompt_frame_runtime | ompt_frame_framepointer
flags all the time, so not something we need to save.

	Jakub