[Bug libgomp/114765] New: linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[Bug libgomp/114765] New: linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[stijn.deweirdt at ugent dot be]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

            Bug ID: 114765
           Summary: linking to libgomp and setting CPU_PROC_BIND causes
                    affinity reset
           Product: gcc
           Version: 13.2.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libgomp
          Assignee: unassigned at gcc dot gnu.org
          Reporter: stijn.deweirdt at ugent dot be
                CC: jakub at gcc dot gnu.org
  Target Milestone: ---

Compiling c code that has no openmp statements, and linking libgomp to it;
running that binary with OMP_PROC_BIND=true causes affinity "reset" to single
core. In output below see differene between "1000000" and "111111" in reported
affinity.

(This may sound odd, but the real origin of this issue comes from some python
code that does an 'import torch' and then all proceeding multiprocessing code
was pinned to single core; (and in an environment where OMP_PROC_BIND was set
to true). in the case of torch, it was the init importing something inked to
openblas which in turn was linked to libgomp)


Code below print affinity (named ompissue.c below)

#define _GNU_SOURCE
#include <assert.h>
#include <sched.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

void print_affinity() {
    cpu_set_t mask;
    long nproc, i;

    if (sched_getaffinity(0, sizeof(cpu_set_t), &mask) == -1) {
        perror("sched_getaffinity");
        assert(false);
    }
    nproc = sysconf(_SC_NPROCESSORS_ONLN);
    printf("sched_getaffinity = ");
    for (i = 0; i < nproc; i++) {
        printf("%d ", CPU_ISSET(i, &mask));
    }
    printf("\n");
}

int main(void) {
    cpu_set_t mask;

    print_affinity();
}


compile and link with "gcc -l gomp ompissue.c"

and then it run:

[stdweird@w011h183 tmp]$ OMP_DISPLAY_ENV=true OMP_PROC_BIND=TRUE ./with_omp 

OPENMP DISPLAY ENVIRONMENT BEGIN
  _OPENMP = '201511'
  [host] OMP_DYNAMIC = 'FALSE'
  [host] OMP_NESTED = 'FALSE'
  [host] OMP_NUM_THREADS = '1'
  [host] OMP_SCHEDULE = 'DYNAMIC'
  [host] OMP_PROC_BIND = 'TRUE'
  [host] OMP_PLACES = '{0},{4},{1},{5},{2},{6},{3},{7}'
  [host] OMP_STACKSIZE = '0'
  [host] OMP_WAIT_POLICY = 'PASSIVE'
  [host] OMP_THREAD_LIMIT = '4294967295'
  [host] OMP_MAX_ACTIVE_LEVELS = '1'
  [host] OMP_NUM_TEAMS = '0'
  [host] OMP_TEAMS_THREAD_LIMIT = '0'
  [all] OMP_CANCELLATION = 'FALSE'
  [all] OMP_DEFAULT_DEVICE = '0'
  [all] OMP_MAX_TASK_PRIORITY = '0'
  [all] OMP_DISPLAY_AFFINITY = 'FALSE'
  [host] OMP_AFFINITY_FORMAT = 'level %L thread %i affinity %A'
  [host] OMP_ALLOCATOR = 'omp_default_mem_alloc'
  [all] OMP_TARGET_OFFLOAD = 'DEFAULT'
OPENMP DISPLAY ENVIRONMENT END
sched_getaffinity = 1 0 0 0 0 0 0 0 
[stdweird@w011h183 tmp]$ OMP_DISPLAY_ENV=true OMP_PROC_BIND=FALSE ./with_omp 

OPENMP DISPLAY ENVIRONMENT BEGIN
  _OPENMP = '201511'
  [host] OMP_DYNAMIC = 'FALSE'
  [host] OMP_NESTED = 'FALSE'
  [host] OMP_NUM_THREADS = '1'
  [host] OMP_SCHEDULE = 'DYNAMIC'
  [host] OMP_PROC_BIND = 'FALSE'
  [host] OMP_PLACES = ''
  [host] OMP_STACKSIZE = '0'
  [host] OMP_WAIT_POLICY = 'PASSIVE'
  [host] OMP_THREAD_LIMIT = '4294967295'
  [host] OMP_MAX_ACTIVE_LEVELS = '1'
  [host] OMP_NUM_TEAMS = '0'
  [host] OMP_TEAMS_THREAD_LIMIT = '0'
  [all] OMP_CANCELLATION = 'FALSE'
  [all] OMP_DEFAULT_DEVICE = '0'
  [all] OMP_MAX_TASK_PRIORITY = '0'
  [all] OMP_DISPLAY_AFFINITY = 'FALSE'
  [host] OMP_AFFINITY_FORMAT = 'level %L thread %i affinity %A'
  [host] OMP_ALLOCATOR = 'omp_default_mem_alloc'
  [all] OMP_TARGET_OFFLOAD = 'DEFAULT'
OPENMP DISPLAY ENVIRONMENT END
sched_getaffinity = 1 1 1 1 1 1 1 1

[Bug libgomp/114765] linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

--- Comment #1 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
That is how it should behave.  Don't use OMP_PROC_BIND=true unless that is what
you want...

[Bug libgomp/114765] linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[stijn.deweirdt at ugent dot be]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

--- Comment #2 from Stijn De Weirdt <stijn.deweirdt at ugent dot be> ---
hi jakub,

apologies for my ignorance, but how is working as expected? i can't make this
up from the description of the variable; and it's unexpected since there is no
openmp code being run (naively then; clearly libgomp does something).

we have an environment where we set OMP_PROC_BIND=true because openmp code
clearly benefits from it; but it also affects code where libgomp is simply
linked to it, but nothing else is done with it.

if i would understand why or what libgomp does, i might be able to find some
workaround.

many thanks,

stijn

[Bug libgomp/114765] linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
(In reply to Stijn De Weirdt from comment #2)
> apologies for my ignorance, but how is working as expected? i can't make
> this up from the description of the variable; and it's unexpected since
> there is no openmp code being run (naively then; clearly libgomp does
> something).

libgomp has library constructors which already do something.

> we have an environment where we set OMP_PROC_BIND=true because openmp code
> clearly benefits from it; but it also affects code where libgomp is simply
> linked to it, but nothing else is done with it.

See the OpenMP standard.  By using that env variable (or similar), you request
that the initial thread binds to a certain CPU/thread (the OMP_PROC_BIND=true
case is less specified than when using say OMP_PLACES and exact rules which
thread binds to which place).

[Bug libgomp/114765] linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[stijn.deweirdt at ugent dot be]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

--- Comment #4 from Stijn De Weirdt <stijn.deweirdt at ugent dot be> ---
hi jakub,


thanks for the explanation. so if i understand this, the main thread/process of
any binary linked with libgomp becomes an OpenMP thread, because of the libgomp
constructor doing something. and thus the OMP_PROC_BIND works as intended.

and what happens with other threads created in the code in such a case? these
are not openmp threads? or those are openmp threads, but related to the
"nesting"/"active levels"?

and can we "delay" the OMP_PROC_BIND safely (eg start the binary without
OMP_PROC_BIND, and "enable" it in the main; i am more thinking in python code
here: so after the import, we set the enviroment with OMP_PROC_BIND=true). will
this cause pinning for openmp threads started afterwards?

anyway, apologies again for my confusion and lack of detailed understanding. i
still find it a bit unexpected ;)

stijn

[Bug libgomp/114765] linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[amonakov at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

Alexander Monakov <amonakov at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |amonakov at gcc dot gnu.org

--- Comment #5 from Alexander Monakov <amonakov at gcc dot gnu.org> ---
Can libgomp defer changing affinity of the initial thread to the launch of the
first parallel region (i.e. change it only at thread pool initialization,
together with new threads)?

[Bug libgomp/114765] linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

--- Comment #6 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
(In reply to Alexander Monakov from comment #5)
> Can libgomp defer changing affinity of the initial thread to the launch of
> the first parallel region (i.e. change it only at thread pool
> initialization, together with new threads)?

The creation of a parallel region is very performance sensitive area, so
slowing it down is undesirable.  But more importantly, if the initial thread
switches binding only at that point, it will mean memory allocated by the
initial thread might no longer be close to that thread.

[Bug libgomp/114765] linking to libgomp and setting CPU_PROC_BIND causes affinity reset

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114765

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://gcc.gnu.org/bugzill
                   |                            |a/show_bug.cgi?id=113698
         Resolution|---                         |INVALID
             Status|UNCONFIRMED                 |RESOLVED

--- Comment #7 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Invalid as explained. Also see PR 113698 .