gfortran, OpenMP and static linking

gfortran, OpenMP and static linking

[Harald Anlauf]

Dear all,

is there a reason that one should not be able to statically link a
Fortran binary that has been compiled with -fopenmp?

A tiny example seems to fail for me with all gfortran versions:

program p
!$omp parallel
!$omp end parallel
end

% gfortran -fopenmp foo.f90 -g -static
% OMP_NUM_THREADS=1 ./a.out

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
[...]

Running under gdb:

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x0000000000405198 in __gthread_mutex_destroy (__mutex=0x72ced8)
    at ../libgcc/gthr-default.h:739
#2  destroy_unit_mutex (u=0x72ce00) at ../../../libgfortran/io/unit.c:252
#3  close_unit_1 (u=0x72ce00, locked=locked@entry=1) at ../../../libgfortran/io/unit.c:743
#4  0x0000000000405202 in _gfortrani_close_units () at ../../../libgfortran/io/unit.c:780
#5  0x000000000044c53c in __libc_csu_fini () at elf-init.c:100
#6  0x0000000000452ef0 in __run_exit_handlers (status=0, listp=0x720630 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:83
#7  0x0000000000452f4a in exit (status=<optimized out>) at exit.c:105
#8  0x000000000044bdc6 in __libc_start_main (main=0x4030bd <main>, argc=1,
    argv=0x7fffffffc348, init=0x44c470 <__libc_csu_init>, fini=0x44c510 <__libc_csu_fini>,
    rtld_fini=0x0, stack_end=0x7fffffffc338) at ../csu/libc-start.c:342
#9  0x0000000000402fba in _start () at ../sysdeps/x86_64/start.S:120


I suspect either a locking issue for unit_root in unit.c, or maybe bad
initialization of it, but haven't stared long enough at the code.
According to git blame almost all related references date back to 2005.

Shall I open a PR against libfortran?  Or is there something I am missing?

Thanks,
Harald

Re: gfortran, OpenMP and static linking

[Bernhard Reutner-Fischer]

On 3 April 2021 20:55:39 CEST, Harald Anlauf via Fortran <fortran@gcc.gnu.org> wrote:
>Dear all,
>
>is there a reason that one should not be able to statically link a
>Fortran binary that has been compiled with -fopenmp?

Maybe Jakub knows more on this.

Not sure if static linking with glibc is still discouraged (it was, at least in former times) or if thread cancellation still is supposed to work in a pure static build.

As said, maybe Jakub can help.
thanks,
>
>A tiny example seems to fail for me with all gfortran versions:
>
>program p
>!$omp parallel
>!$omp end parallel
>end
>
>% gfortran -fopenmp foo.f90 -g -static
>% OMP_NUM_THREADS=1 ./a.out
>
>Program received signal SIGSEGV: Segmentation fault - invalid memory
>reference.
>[...]
>
>Running under gdb:
>
>Program received signal SIGSEGV, Segmentation fault.
>0x0000000000000000 in ?? ()
>(gdb) bt
>#0  0x0000000000000000 in ?? ()
>#1  0x0000000000405198 in __gthread_mutex_destroy (__mutex=0x72ced8)
>    at ../libgcc/gthr-default.h:739
>#2  destroy_unit_mutex (u=0x72ce00) at
>../../../libgfortran/io/unit.c:252
>#3  close_unit_1 (u=0x72ce00, locked=locked@entry=1) at
>../../../libgfortran/io/unit.c:743
>#4  0x0000000000405202 in _gfortrani_close_units () at
>../../../libgfortran/io/unit.c:780
>#5  0x000000000044c53c in __libc_csu_fini () at elf-init.c:100
>#6  0x0000000000452ef0 in __run_exit_handlers (status=0, listp=0x720630
><__exit_funcs>,
>run_list_atexit=run_list_atexit@entry=true,
>run_dtors=run_dtors@entry=true) at exit.c:83
>#7  0x0000000000452f4a in exit (status=<optimized out>) at exit.c:105
>#8  0x000000000044bdc6 in __libc_start_main (main=0x4030bd <main>,
>argc=1,
>argv=0x7fffffffc348, init=0x44c470 <__libc_csu_init>, fini=0x44c510
><__libc_csu_fini>,
>    rtld_fini=0x0, stack_end=0x7fffffffc338) at ../csu/libc-start.c:342
>#9  0x0000000000402fba in _start () at ../sysdeps/x86_64/start.S:120
>
>
>I suspect either a locking issue for unit_root in unit.c, or maybe bad
>initialization of it, but haven't stared long enough at the code.
>According to git blame almost all related references date back to 2005.
>
>Shall I open a PR against libfortran?  Or is there something I am
>missing?
>
>Thanks,
>Harald

Re: gfortran, OpenMP and static linking

[Bernhard Reutner-Fischer]

Harald,

On Sat, 03 Apr 2021 23:24:52 +0200
Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> wrote:

> On 3 April 2021 20:55:39 CEST, Harald Anlauf via Fortran <fortran@gcc.gnu.org> wrote:
> >Dear all,
> >
> >is there a reason that one should not be able to statically link a
> >Fortran binary that has been compiled with -fopenmp?  
> 
> Maybe Jakub knows more on this.
> 
> Not sure if static linking with glibc is still discouraged (it was, at least in former times) or if thread cancellation still is supposed to work in a pure static build.

You can try to get away with linking libpthread in like below.
However, glibc is (still) not supposed to be linked statically (don't
ask me, i didn't write it).
HTH,

$ cat omp.f90 
use omp_lib
!$omp parallel
  write(*,*) "thread ", omp_get_thread_num()
!$omp end parallel
end
$ gfortran -o omp -fopenmp omp.f90 -static -Wl,--whole-archive -lpthread -Wl,--no-whole-archive && OMP_NUM_THREADS=2 ./omp
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/10/libgomp.a(target.o): in function `gomp_target_init':
(.text+0x358): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
 thread            0
 thread            1

Re: gfortran, OpenMP and static linking

[Harald Anlauf]

Hi Bernhard,

> $ gfortran -o omp -fopenmp omp.f90 -static -Wl,--whole-archive -lpthread -Wl,--no-whole-archive && OMP_NUM_THREADS=2 ./omp

that does indeed do the trick!

As suggested in your other mail, I should ask Jakub for his opinion.

(Statically compiling/linking on the same machine with the Intel compiler
seems to simply work without similar issues. )

Thanks,
Harald

gfortran, OpenMP and static linking

[Andrew Benson]

I've been following the fix proposed by Bernhard in this thread:

https://gcc.gnu.org/pipermail/fortran/2021-April/055907.html

to allow static linking of Fortran codes compiled with -fopenmp. (I know that 
static linking of OpenMP codes isn't guaranteed to work, but this fix has 
worked well for me anyway.)

This fix seems to no longer work with glibc >= 2.34 - I think because the 
functionality of libpthread has been integrated into libc:

https://sourceware.org/pipermail/libc-alpha/2021-August/129718.html

When compiling the simple test code from that thread with glibc 2.34 or newer 
I get a segfault:

$ cat omp.f90 
use omp_lib
!$omp parallel
  write(*,*) "thread ", omp_get_thread_num()
!$omp end parallel
end
$ gfortran -o omp -fopenmp omp.f90 -static -Wl,--whole-archive -lpthread -
Wl,--no-whole-archive && OMP_NUM_THREADS=2 ./omp
./../../gcc-13-source/gcc-13-20220710/libgomp/config/linux/../../allocator.c:
102: warning: Using 'dlopen' in statically linked applications requires at 
runtime the shared libraries from the glibc version used for linking
./../../gcc-13-source/gcc-13-20220710/libgomp/oacc-profiling.c:137: warning: 
Using 'dlopen' in statically linked applications requires at runtime the 
shared libraries from the glibc version used for linking
 thread            0
 thread            1

Program received signal SIGSEGV: Segmentation fault - invalid memory 
reference.

Backtrace for this error:
#0  0x467b0f in ???
#1  0x0 in ???
Segmentation fault (core dumped)

Tracking this down in gdb it seems that pthread_mutex_destroy() is called, but 
that symbol is null. 

From comment #9 in this PR:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58909#c9

I figured out a workaround, which is to create a C file with this content:

#include "pthread.h"

#define nullptr ((void*)0)

void pthread_workaround() {
  pthread_mutex_destroy((pthread_mutex_t *) nullptr);
}

which defines a function pthread_workaround() which calls 
pthread_mutex_destroy(), but which itself it never called. Compiling that C 
code, and linking with the Fortran allows the test code to run successfully.

$ gcc -c -o fix.o -fopenmp fix.c
$ gfortran -c -o omp.o -fopenmp omp.f90
$ gfortran -o omp -fopenmp -static omp.o fix.o
./../../gcc-13-source/gcc-13-20220710/libgomp/config/linux/../../allocator.c:
102: warning: Using 'dlopen' in statically linked applications requires at 
runtime the shared libraries from the glibc version used for linking
./../../gcc-13-source/gcc-13-20220710/libgomp/oacc-profiling.c:137: warning: 
Using 'dlopen' in statically linked applications requires at runtime the 
shared libraries from the glibc version used for linking
$ OMP_NUM_THREADS=2 ./omp
 thread            0
 thread            1

My understanding of glibc, pthreads, etc. etc. is insufficient to know if this 
is a good and/or safe workaround, but I figured I'd share it here in case:

a) it's useful to anyone;
b) it's indicative of a bug in gfortran;
c) anyone has a more elegant solution!

-Andrew

-- 

* Andrew Benson: https://abensonca.github.io

* Galacticus: https://bitbucket.org/abensonca/galacticus