[Bug libgomp/44833] New: unexpected thread binding for openmp

[Bug libgomp/44833] New: unexpected thread binding for openmp

[jv244 at cam dot ac dot uk]

if I use thread binding in the following way:

export OMP_NUM_THREADS=8
export GOMP_CPU_AFFINITY="0 1 2 3 4 5 6 7"

for code that has three parallel regions:

!$omp parallel default(private) shared(bindings,n_thread)
!$omp end parallel
!$omp parallel default(private) shared(bindings,n_thread) num_threads(2)
!$omp end parallel
!$omp parallel default(private) shared(bindings,n_thread)
!$omp end parallel

I get the following:

> ./a.out
 Entering first parallel region all threads
bindings are 0 1 2 3 4 5 6 7
 Entering second parallel region using two threads
bindings are 0 1
 Entering third parallel region using all threads
bindings are 0 1 0 1 2 3 4 5

so it the last parallel section 2 times 2 threads bind to the same CPU (this is
also what is effectively happing looking at the CPU usage in top).

this seems to be happening for gcc 4.3 - 4.6

The testcase will be attached.


-- 
           Summary: unexpected thread binding for openmp
           Product: gcc
           Version: 4.3.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libgomp
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: jv244 at cam dot ac dot uk


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jv244 at cam dot ac dot uk]

------- Comment #1 from jv244 at cam dot ac dot uk  2010-07-06 08:03 -------
Created an attachment (id=21099)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=21099&action=view)
testcase part 1

C code to report thread binding


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jv244 at cam dot ac dot uk]

------- Comment #2 from jv244 at cam dot ac dot uk  2010-07-06 08:05 -------
Created an attachment (id=21100)
 --> (http://gcc.gnu.org/bugzilla/attachment.cgi?id=21100&action=view)
fortran testcase

build & run testcase as :

gfortran -fopenmp test.f90 get_affinity.c
export OMP_NUM_THREADS=8
export GOMP_CPU_AFFINITY="0 1 2 3 4 5 6 7"
./a.out


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jakub at gcc dot gnu dot org]

------- Comment #3 from jakub at gcc dot gnu dot org  2010-07-06 08:21 -------
That's how GOMP_CPU_AFFINITY works - it is a round-robin assignment of CPUs
upon thread creation, and the first two threads are never recreated in this
case.
Currently there is no tracking on how many threads bind to which CPU etc.
We'd need to track that in some data structure and choose CPU with lowest
number of assigned threads or something.


-- 

jakub at gcc dot gnu dot org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |openmp


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jv244 at cam dot ac dot uk]

------- Comment #4 from jv244 at cam dot ac dot uk  2010-07-06 10:01 -------
(In reply to comment #3)
> That's how GOMP_CPU_AFFINITY works - it is a round-robin assignment of CPUs
> upon thread creation, and the first two threads are never recreated in this
> case.
> Currently there is no tracking on how many threads bind to which CPU etc.
> We'd need to track that in some data structure and choose CPU with lowest
> number of assigned threads or something.

oops... this really is a surprise, is there a way that these 6 threads are not
destroyed entering the parallel region that asks for 2 threads. That should fix
the problem. Other compilers do the expected thing, for example intel:

ifort -openmp test.f90 get_affinity.o
export OMP_NUM_THREADS=8
export KMP_AFFINITY="proclist=[0,1,2,3,4,5,6,7]"
>  ./a.out
OMP: Warning #63: KMP_AFFINITY: proclist specified, setting affinity type to
"explicit".
 Entering first parallel region all threads
bindings are 0 1 2 3 4 5 6 7
 Entering second parallel region using two threads
bindings are 0 1
 Entering third parallel region using all threads
bindings are 0 1 2 3 4 5 6 7


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jv244 at cam dot ac dot uk]

------- Comment #5 from jv244 at cam dot ac dot uk  2010-07-06 11:32 -------
I also checked the pgi and cray compilers, they all lead to iforts thread
binding.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jv244 at cam dot ac dot uk]

------- Comment #6 from jv244 at cam dot ac dot uk  2010-07-08 06:24 -------
I have also tried to run the testcase with 

export OMP_WAIT_POLICY=active
export GOMP_SPINCOUNT=infinity

which I would assume to keep the threads alive, and so there would be no need
to create these new threads (the number of active threads is never more than
8), and still this strange binding persists. Jakub, is there any practical way
to avoid recreating these threads ?  


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jakub at gcc dot gnu dot org]

------- Comment #7 from jakub at gcc dot gnu dot org  2010-07-08 07:54 -------
Currently libgomp terminates all threads immediately, unless mandated by the
standard (i.e. in non-nested case the threads need to be kept around until next
omp parallel).

Yes, it would be possible to keep other threads around, the question is how
many and how long, at which point start to destroy them.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833

[Bug libgomp/44833] unexpected thread binding for openmp

[jv244 at cam dot ac dot uk]

------- Comment #8 from jv244 at cam dot ac dot uk  2010-07-08 08:55 -------
(In reply to comment #7)
> Yes, it would be possible to keep other threads around, the question is how
> many and how long, at which point start to destroy them.

I can test what other implementations are doing, but I suspect that for the
non-nested case, OMP_NUM_THREADS will be created and never destroyed. At least
for HPC applications this seems to make sense.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44833