[Bug libgomp/52303] New: libgomp leaves threads lying around that cause trouble if the program is later fork()'d

[Bug libgomp/52303] New: libgomp leaves threads lying around that cause trouble if the program is later fork()'d

[sdh4 at iastate dot edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52303

             Bug #: 52303
           Summary: libgomp leaves threads lying around that cause trouble
                    if the program is later fork()'d
    Classification: Unclassified
           Product: gcc
           Version: 4.6.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libgomp
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: sdh4@iastate.edu


Libgomp creates a thread pool on each thread with a parallel section. This pool
silently hangs around even when no longer in the parallel section. 

If the process is later fork()'d and the child process subsequently enters a
parallel section, the child process will deadlock waiting on threads that do
not exist (fork() does not duplicate threads). 

The pthreads library provides a mechanism to avoid problems such as this.
Specifically (quoting the manpage for pthread_atfork()): 
       The  pthread_atfork() function provides multi-threaded libraries with a
       means to protect themselves from  innocent  application  programs  that
       call fork(), and it provides multi-threaded application programs with a
       standard mechanism for protecting themselves from  fork()  calls  in  a
       library routine or the application itself.

libgomp should call pthread_atfork() with team.c/gomp_free_thread() (or a
wrapper) as the "prepare" parameter so as to destroy the threadpool before
fork() is executed. 

The problem can be worked around by placing the parallel section in a
sub-thread that exits before fork() is called. libgomp automatically destroys
the thread-pool of a thread that exits.

[Bug libgomp/52303] libgomp leaves threads lying around that cause trouble if the program is later fork()'d

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52303

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
                 CC|                            |jakub at gcc dot gnu.org
         Resolution|                            |INVALID

--- Comment #1 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-02-18 10:00:31 UTC ---
That is not valid in POSIX, after fork from multi-threaded process (which is
any process that ever executed #pragma omp parallel too), POSIX lists just a
few functions that you may call after fork before you call _exit or exec*.
#pragma omp parallel is definitely not something you can do after fork from
multi-threaded programs.

[Bug libgomp/52303] libgomp leaves threads lying around that cause trouble if the program is later fork()'d

[sdh4 at iastate dot edu]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52303

Steve Holland <sdh4 at iastate dot edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |UNCONFIRMED
         Resolution|INVALID                     |

--- Comment #2 from Steve Holland <sdh4 at iastate dot edu> 2012-02-18 22:56:16 UTC ---
Ah.... But once it's outside a parallel section, what reason does the
programmer have to think it's still multithreaded?

To quote the OpenMP 3.1 spec: "Only the master thread resumes
execution beyond the end of the parallel construct."

POSIX at minimum suggests that as long as the process has only one remaining
thread, fork() followed by other routines is fine: 
  > A process shall be created with a single thread. If a multi-threaded 
  > process calls fork(), the new process shall contain a replica of the 
  > calling thread and its entire address space, possibly including the 
  > states of mutexes and other resources. Consequently, to avoid errors, 
  > the child process may only execute async-signal-safe operations until 
  > such time as one of the exec functions is called. Fork handlers may 
  > be established by means of the pthread_atfork() function in order to 
  > maintain application invariants across fork() calls.

So why shouldn't it be OK to fork outside a parallel section? 

One important candidate use case (not what I'm doing) would be on clustered HPC
systems that support process migration. The program would start out on a single
process and fork() creates processes that get migrated across the cluster. Each
process then uses OpenMP to distribute its load across the local cores. Any
program that happens to run any parallel code (e.g. to solve for high-level
parameters of the problem) before the forking would deadlock in the first
parallel section after the fork. 

My use case is using Python as a high-level scripting language with loop
parallelization a la http://packages.python.org/joblib/parallel.html. If you 
run some C extension module that uses OpenMP and then start a parallel loop
that also ends up running code that uses OpenMP, you again have a deadlock. 

Perhaps the C code that uses OpenMP should clean up the threads before
returning to Python, but there's no API for this. 

This is an easy enough fix, isn't it?  pthread_atfork() exists exactly for this
situation, where a library leaves threads hanging around that need to be
cleaned up in case of fork().

[Bug libgomp/52303] libgomp leaves threads lying around that cause trouble if the program is later fork()'d

[jakub at gcc dot gnu.org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52303

Jakub Jelinek <jakub at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|                            |INVALID

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> 2012-02-19 18:01:58 UTC ---
Preserving the threads after the parallel region finishes is basically required
by the standard, as a new parallel region with the same number of threads needs
to preserve threadprivate data.  So even after a parallel region is left, your
process is still multi-threaded.

[Bug libgomp/52303] libgomp leaves threads lying around that cause trouble if the program is later fork()'d

[olivier.grisel at ensta dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52303

Olivier Grisel <olivier.grisel at ensta dot org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |olivier.grisel at ensta dot org

--- Comment #4 from Olivier Grisel <olivier.grisel at ensta dot org> ---
Wouldn't it be possible to make the libgomp runtime record the pid of the
process who initialized the openmp thread pool and then whenever a parallel
section is reached by the process to check whether the pid of the current
process matches the pid that initialized the thread pool? If not a new init
could be triggered at that point to make it possible to have a fork-child
process use openmp sections in turn transparently.

Alternatively it would be nice to have access to a public function to shutdown
("de-initialize") a thread pool prior to a fork.

[Bug libgomp/52303] libgomp leaves threads lying around that cause trouble if the program is later fork()'d

[olivier.grisel at ensta dot org]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52303

--- Comment #5 from Olivier Grisel <olivier.grisel at ensta dot org> ---
The second option is basically was is being asked for at the end of this
section:

  http://bisqwit.iki.fi/story/howto/openmp/#OpenmpAndFork