[Bug libgomp/98738] New: task-detach-6.f90 hangs intermittently

[Bug libgomp/98738] New: task-detach-6.f90 hangs intermittently

[iii at linux dot ibm.com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

            Bug ID: 98738
           Summary: task-detach-6.f90 hangs intermittently
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libgomp
          Assignee: unassigned at gcc dot gnu.org
          Reporter: iii at linux dot ibm.com
                CC: jakub at gcc dot gnu.org
  Target Milestone: ---

I'm currently on commit 2e43880dbd4c.  Building task-detach-6.exe and running
it in a loop eventually leads to a hang (might take a while, during the last
run it happened after 7k runs):

gcc/build/x86_64-pc-linux-gnu/libgomp/testsuite$ while true; do
LD_LIBRARY_PATH=../../../../install/lib64 ./task-detach-6.exe; echo -n .; done

I first spotted this on s390 and then checked on x86_64; the issue is
reproducible on both.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[iii at linux dot ibm.com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #1 from Ilya Leoshkevich <iii at linux dot ibm.com> ---
I realized I didn't post the command line I used to build task-detach-6.exe
(there are multiple variants of this test); here it is:

gcc/build/x86_64-pc-linux-gnu/libgomp/testsuite$ ../../../../build/./gcc/xgcc
-B../../../../build/./gcc/ -B../../../../install/x86_64-pc-linux-gnu/bin/
-B../../../../install/x86_64-pc-linux-gnu/lib/ -isystem
../../../../install/x86_64-pc-linux-gnu/include -isystem
../../../../install/x86_64-pc-linux-gnu/sys-include -fchecking=1
../../../../libgomp/testsuite/libgomp.fortran/task-detach-6.f90
-B../../../../build/x86_64-pc-linux-gnu/./libgomp/
-B../../../../build/x86_64-pc-linux-gnu/./libgomp/.libs
-I../../../../build/x86_64-pc-linux-gnu/./libgomp
-I../../../../libgomp/testsuite/../../include
-I../../../../libgomp/testsuite/.. -fmessage-length=0
-fno-diagnostics-show-caret -fdiagnostics-color=never -fopenmp
-B../../../../build/x86_64-pc-linux-gnu/./libgomp/../libquadmath/.libs/ -O1
-B../../../../build/x86_64-pc-linux-gnu/./libgomp/../libgfortran/.libs
-fintrinsic-modules-path=../../../../build/x86_64-pc-linux-gnu/./libgomp
-L../../../../build/x86_64-pc-linux-gnu/./libgomp/.libs
-L../../../../build/x86_64-pc-linux-gnu/./libgomp/../libquadmath/.libs/
-L../../../../build/x86_64-pc-linux-gnu/./libgomp/../libgfortran/.libs
-lgfortran -foffload=-lgfortran -lquadmath -lm -o ./task-detach-6.exe

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

H.J. Lu <hjl.tools at gmail dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |NEW
                 CC|                            |hjl.tools at gmail dot com
   Last reconfirmed|                            |2021-01-19
     Ever confirmed|0                           |1

--- Comment #2 from H.J. Lu <hjl.tools at gmail dot com> ---
Also happen to me on Linux/x86-64.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #3 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
Also on Darwin:

WARNING: program timed out.
FAIL: libgomp.c/../libgomp.c-c++-common/task-detach-6.c execution test
WARNING: program timed out.
FAIL: libgomp.fortran/task-detach-6.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
WARNING: program timed out.
FAIL: libgomp.fortran/task-detach-6.f90   -Os  execution test
WARNING: program timed out.
FAIL: libgomp.fortran/task-detach-6.f90   -g -O3 -fwhole-program -flto 
execution test

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[ro at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

Rainer Orth <ro at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ro at gcc dot gnu.org
             Target|s390x, x86_64-*-*           |s390x, x86_64-*-*,
                   |                            |*-*-solaris*
   Target Milestone|---                         |11.0

--- Comment #4 from Rainer Orth <ro at gcc dot gnu.org> ---
I'm seeing the same on Solaris (both sparc and x86) for both
libgomp.c-c++-common/task-detach-6.c
and libgomp.fortran/task-detach-6.f90.  When running the C test in a loop,
about
1 out of 10 times it hangs.

On Linux/x86_64, its way rarer (1 out of 1000 or less), however when
configuring
with --disable-linux-futex the frequency is similar to what I see on Solaris.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
At least the
firstprivate(detach_event1, detach_event2)
on parallel look incorrect, the vars are uninitialized at that point, so
copying those copies uninitialized values.
private(detach_event1, detach_event2)
looks more correct.
But that shouldn't make the program wrong.
Kwok, can you just for debugging change 
task_fulfilled_p to return false; ?
At least in my understanding of your code, it is just an optimization, because
whether the omp_fullfill_event is called nanosecond before or after the
task_fulfilled_p check shouldn't make a difference on the correctness of the
program, and perhaps if it is return false; it will make the hangs
reproduceable.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[kcy at codesourcery dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #6 from Kwok Yeung <kcy at codesourcery dot com> ---
It seems to be the '#pragma omp taskwait' inside the parallel region that is
causing the intermittent hang - if you add it to task-detach-5.c, the hang also
happens there (i.e. the target construct is not the problem). I think I added
the taskwait because otherwise it always hung when the region was offloaded to
a Nvidia accelerator.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #7 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Seems making task_fulfilled_p return always false makes the task-detach-6.c
hangs 100% reproduceable.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #8 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
I think omp_fullfill_event needs to do much more than it does.
Just look e.g. at what gomp_target_task_completion does.  The function needs to
find out in what state the task is (I'm afraid it needs to take the
team->task_lock around that checking (if any)), if the task hasn't finished
yet, it shouldn't need anything but sem_post but if it has already finished and
it is just waiting for the completion event, it needs to handle taskwait,
depend wait, taskgroup waits and barrier waits.
Sorry for not catching that during patch review.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[kcy at codesourcery dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #9 from Kwok Yeung <kcy at codesourcery dot com> ---
I have a patch for this now. In addition to posting other semaphores in
omp_fulfill_event, GOMP_taskwait also needs to be made aware of detached tasks.
Also, task_running_count should not include completed tasks that are waiting
for event completion, since these are not really still running, and may lead to
a failure to wake up the barrier thread when required.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #10 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
I've been wondering whether it wouldn't be best to pass through
omp_eventhandle_t  the pointer to the gomp_task rather than some particular
semaphore and in omp_fulfill_event then decide based on that task, one case
would be if it is the included task, then it should post a semaphore and be
done with it, otherwise (not included task), it should find the corresponding
team (which might not be necessary the omp_fulfill_event's task), lock the
task_lock and figure out in which of the many cases the task is in, if not yet
finished, perhaps just clear detach bool and be done with it, if in the state
after that, figure out what is waiting for that task and deal with that.
Not sure if a per-task semaphore is what we want in that case, e.g. we could
have a taskwait, taskgroup end or barrier waiting for multiple tasks with
detach clause and if that is the only thing they might be waiting on, then
those should be sleeping on taskwait_sem, taskgroup_sem or on the barrier, so
depending on what they are sleeping on that should be awaken.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[dominiq at lps dot ens.fr]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

Dominique d'Humieres <dominiq at lps dot ens.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |seurer at gcc dot gnu.org

--- Comment #11 from Dominique d'Humieres <dominiq at lps dot ens.fr> ---
*** Bug 98808 has been marked as a duplicate of this bug. ***

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[cvs-commit at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #12 from CVS Commits <cvs-commit at gcc dot gnu.org> ---
The master branch has been updated by Kwok Yeung <kcy@gcc.gnu.org>:

https://gcc.gnu.org/g:d656bfda2d8316627d0bbb18b10954e6aaf3c88c

commit r11-7407-gd656bfda2d8316627d0bbb18b10954e6aaf3c88c
Author: Kwok Cheung Yeung <kcy@codesourcery.com>
Date:   Thu Jan 21 05:38:47 2021 -0800

    openmp: Fix intermittent hanging of task-detach-6 libgomp tests [PR98738]

    This adds support for the task detach clause to taskwait and taskgroup, and
    simplifies the handling of the detach clause by moving most of the extra
    handling required for detach tasks to omp_fulfill_event.

    2021-02-25  Kwok Cheung Yeung  <kcy@codesourcery.com>
                Jakub Jelinek  <jakub@redhat.com>

            libgomp/

            PR libgomp/98738
            * libgomp.h (enum gomp_task_kind): Add GOMP_TASK_DETACHED.
            (struct gomp_task): Replace detach and completion_sem fields with
            union containing completion_sem and detach_team.  Add deferred_p
            field.
            (struct gomp_team): Remove task_detach_queue.
            * task.c: Include assert.h.
            (gomp_init_task): Initialize deferred_p and completion_sem fields.
            Rearrange initialization order of fields.
            (task_fulfilled_p): Delete.
            (GOMP_task): Use address of task as the event handle.  Remove
            initialization of detach field.  Initialize deferred_p field.
            Use automatic local for completion_sem.  Initialize detach_team
field
            for deferred tasks.
            (gomp_barrier_handle_tasks): Remove handling of task_detach_queue.
            Set kind of suspended detach task to GOMP_TASK_DETACHED and
            decrement task_running_count.  Move finish_cancelled block out of
            else branch.  Relocate call to gomp_team_barrier_done.
            (GOMP_taskwait): Handle tasks with completion events that have not
            been fulfilled.
            (GOMP_taskgroup_end): Likewise.
            (omp_fulfill_event): Use address of task as event handle.  Post to
            completion_sem for undeferred tasks.  Clear detach_team if task
            has not finished.  For finished tasks, handle post-execution tasks,
            call gomp_team_barrier_wake if necessary, and free task.
            * team.c (gomp_new_team): Remove initialization of
task_detach_queue.
            (free_team): Remove free of task_detach_queue.
            * testsuite/libgomp.c-c++-common/task-detach-1.c: Fix formatting.
            * testsuite/libgomp.c-c++-common/task-detach-2.c: Fix formatting.
            * testsuite/libgomp.c-c++-common/task-detach-3.c: Fix formatting.
            * testsuite/libgomp.c-c++-common/task-detach-4.c: Fix formatting.
            * testsuite/libgomp.c-c++-common/task-detach-5.c: Fix formatting.
            Change data-sharing of detach events on enclosing parallel to
private.
            * testsuite/libgomp.c-c++-common/task-detach-6.c: Likewise.  Remove
            taskwait directive.
            * testsuite/libgomp.c-c++-common/task-detach-7.c: New.
            * testsuite/libgomp.c-c++-common/task-detach-8.c: New.
            * testsuite/libgomp.c-c++-common/task-detach-9.c: New.
            * testsuite/libgomp.c-c++-common/task-detach-10.c: New.
            * testsuite/libgomp.c-c++-common/task-detach-11.c: New.
            * testsuite/libgomp.fortran/task-detach-1.f90: Fix formatting.
            * testsuite/libgomp.fortran/task-detach-2.f90: Fix formatting.
            * testsuite/libgomp.fortran/task-detach-3.f90: Fix formatting.
            * testsuite/libgomp.fortran/task-detach-4.f90: Fix formatting.
            * testsuite/libgomp.fortran/task-detach-5.f90: Fix formatting.
            Change data-sharing of detach events on enclosing parallel to
private.
            * testsuite/libgomp.fortran/task-detach-6.f90: Likewise.  Remove
            taskwait directive.
            * testsuite/libgomp.fortran/task-detach-7.f90: New.
            * testsuite/libgomp.fortran/task-detach-8.f90: New.
            * testsuite/libgomp.fortran/task-detach-9.f90: New.
            * testsuite/libgomp.fortran/task-detach-10.f90: New.
            * testsuite/libgomp.fortran/task-detach-11.f90: New.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[tschwinge at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

Thomas Schwinge <tschwinge at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tschwinge at gcc dot gnu.org
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |kcy at codesourcery dot com

--- Comment #13 from Thomas Schwinge <tschwinge at gcc dot gnu.org> ---
Kwok, it seems -- at least in my testing -- that your latest commit
d656bfda2d8316627d0bbb18b10954e6aaf3c88c "openmp: Fix intermittent hanging of
task-detach-6 libgomp tests [PR98738]" has broken things with nvptx offloading
enabled: because of hanging
'libgomp.c/../libgomp.c-c++-common/task-detach-6.c', I manually terminated
testing after:

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    24507 thomas    20   0 9157400 130444 122980 R 100.0  0.4  41:13.59
./task-detach-6.exe

..., and another run after:

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    32365 thomas    20   0 9159472 140040 128136 R 100.0  0.4  59:05.48
./task-detach-6.exe

I had 100 % GPU utilization in this state.

Is there something wrong (are you or anyone reproducing that with nvptx
offloading?), or is something wrong on my side?

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #14 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
https://gcc.gnu.org/pipermail/gcc-patches/2021-February/565740.html ?

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[kcy at codesourcery dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #15 from Kwok Yeung <kcy at codesourcery dot com> ---
(In reply to Thomas Schwinge from comment #13)
> Kwok, it seems -- at least in my testing -- that your latest commit
> d656bfda2d8316627d0bbb18b10954e6aaf3c88c "openmp: Fix intermittent hanging
> of task-detach-6 libgomp tests [PR98738]" has broken things with nvptx
> offloading enabled: because of hanging
> 'libgomp.c/../libgomp.c-c++-common/task-detach-6.c'

I noted this hang at:

https://gcc.gnu.org/pipermail/gcc-patches/2021-February/565740.html

It does not seem to be an issue with the detach support though, as this trivial
testcase will also hang.

int main (void)
{
#pragma omp target
   #pragma omp parallel
     #pragma omp task
       ;
}

I have confirmed that this behaviour occurs even before my original patch for
implementing the detach clause, and it does not occur with GCN offloading.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[tschwinge at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #16 from Thomas Schwinge <tschwinge at gcc dot gnu.org> ---
Ugh.  :-( Where are we tracking this, and is anyone working on this?  It's
clearly not useful to have (nvptx offloading) testing to run into known
TIMEOUTs?

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[tschwinge at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

--- Comment #17 from Thomas Schwinge <tschwinge at gcc dot gnu.org> ---
I've filed PR99555 "[OpenMP/nvptx] Execution-time hang for simple nested OpenMP
'target'/'parallel'/'task' constructs".

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|11.0                        |11.2

--- Comment #18 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
GCC 11.1 has been released, retargeting bugs to GCC 11.2.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|11.2                        |11.3

--- Comment #19 from Richard Biener <rguenth at gcc dot gnu.org> ---
GCC 11.2 is being released, retargeting bugs to GCC 11.3

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

Richard Biener <rguenth at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|11.3                        |11.4

--- Comment #20 from Richard Biener <rguenth at gcc dot gnu.org> ---
GCC 11.3 is being released, retargeting bugs to GCC 11.4.

[Bug libgomp/98738] task-detach-6.f90 hangs intermittently

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98738

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|11.4                        |11.5

--- Comment #21 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
GCC 11.4 is being released, retargeting bugs to GCC 11.5.