[Bug libgomp/113698] New: GNU OpenMP with OMP_PROC_BIND alters thread affinity in a way that negatively affects performance

[Bug libgomp/113698] New: GNU OpenMP with OMP_PROC_BIND alters thread affinity in a way that negatively affects performance

[kugan at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113698

            Bug ID: 113698
           Summary: GNU OpenMP with OMP_PROC_BIND alters thread affinity
                    in a way that negatively affects performance
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: libgomp
          Assignee: unassigned at gcc dot gnu.org
          Reporter: kugan at gcc dot gnu.org
                CC: jakub at gcc dot gnu.org
  Target Milestone: ---

Created attachment 57275
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=57275&action=edit
testcase

When OMP_PROC_BIND=true it seems gomp set the affinity even before main()
starts. In particular, the main thread gets affinity 0x1 (i.e. pinned to the
first core). For the attached, I get

$ OMP_NUM_THREADS=72 ./a.out
[main thread affinity right after main()]. tid:ffffae511020
aff:0000000000000000.0000000000000000.000000000000ffff.ffffffffffffffff
duration: 402.949 msec

$ OMP_PROC_BIND=true OMP_NUM_THREADS=72 ./a.out
[main thread affinity right after main()]. tid:fffdded50020
aff:0000000000000000.0000000000000000.0000000000000000.0000000000000001
duration: 7879.59 msec

$ OMP_PROC_BIND=true OMP_NUM_THREADS=72 ./a.out
[main thread affinity right after main()]. tid:ffffae54c020
aff:0000000000000000.0000000000000000.0000000000000000.0000000000000001
duration: 311219 msec

Compiler options used:
gcc -O0 -fopenmp repro.c

gcc -v:


Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/11/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs
--enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr
--with-gcc-major-version-only --program-suffix=-11
--program-prefix=aarch64-linux-gnu- --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes
--with-default-libstdcxx-abi=new --enable-gnu-unique-object
--disable-libquadmath --disable-libquadmath-support --enable-plugin
--enable-default-pie --with-system-zlib --enable-libphobos-checking=release
--with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch
--enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release
--build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu
--with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)

[Bug libgomp/113698] GNU OpenMP with OMP_PROC_BIND alters thread affinity in a way that negatively affects performance

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113698

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
https://www.openmp.org/spec-html/5.0/openmpse52.html

> Otherwise, the execution environment should not move OpenMP threads between OpenMP places, thread affinity is enabled, and __the initial thread is bound to the first place in the OpenMP place list prior to the first active parallel region__.

Hmm. Maybe LLVM's libomp does not follow the openmp spec.

[Bug libgomp/113698] GNU OpenMP with OMP_PROC_BIND alters thread affinity in a way that negatively affects performance

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113698

--- Comment #2 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Actually I misread the testcase, because it looks like LLVM's libomp follows
the same as GCC's.

And it looks like the OpenMP spec is specific about that behavior too.

[Bug libgomp/113698] GNU OpenMP with OMP_PROC_BIND alters thread affinity in a way that negatively affects performance

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113698

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
You can use GOMP_CPU_AFFINITY or different setting of OMP_PROC_BIND together
with OMP_PLACES to bind the initial thread to different CPU.
While the OpenMP spec allows moving the initial thread from unspecified place
to the requested one only before the first active parallel region, doing that
there would add to the overhead of #pragma omp parallel and what exactly the
application wants is hard to guess, whether the allocations done in the initial
thread prior to the first parallel are meant for the initial thread or other
threads too.
And, if you just add an empty #pragma omp parallel at the start of main, then
OpenMP requires the thread to be bound there.
There are OpenMP allocators one can use and request specific thread binding for
the allocations, or perform the allocations from the threads.

[Bug libgomp/113698] GNU OpenMP with OMP_PROC_BIND alters thread affinity in a way that negatively affects performance

[kugan at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113698

--- Comment #4 from kugan at gcc dot gnu.org ---
Thanks for looking into this. The main reason we ere seeing performance issue
turned out to be due to glibc malloc issue in
https://sourceware.org/bugzilla/show_bug.cgi?id=30945

[Bug libgomp/113698] GNU OpenMP with OMP_PROC_BIND alters thread affinity in a way that negatively affects performance

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113698

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|UNCONFIRMED                 |RESOLVED
         Resolution|---                         |MOVED

--- Comment #5 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
Moved and looks to be fixed