[Bug libfortran/113005] New: 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[Bug libfortran/113005] New: 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[tschwinge at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

            Bug ID: 113005
           Summary: 'libgomp.fortran/rwlock_1.f90',
                    'libgomp.fortran/rwlock_3.f90' execution test timeouts
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Keywords: testsuite-fail
          Severity: normal
          Priority: P3
         Component: libfortran
          Assignee: unassigned at gcc dot gnu.org
          Reporter: tschwinge at gcc dot gnu.org
                CC: hjl at gcc dot gnu.org
  Target Milestone: ---
            Target: powerpc64le-linux-gnu

On several of our "big iron" powerpc64le-linux-gnu systems, I'm seeing the new
test cases 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' run
into execution test timeouts (300 s).  Those were added in commit
r14-6425-gb806c88fab3f9c6833563f9a44b608dd5dd14de9 "libgfortran: Replace mutex
with rwlock".

    PASS: libgomp.fortran/rwlock_1.f90   -O0  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_1.f90   -O0  execution test
    PASS: libgomp.fortran/rwlock_1.f90   -O1  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_1.f90   -O1  execution test
    PASS: libgomp.fortran/rwlock_1.f90   -O2  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_1.f90   -O2  execution test
    PASS: libgomp.fortran/rwlock_1.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  (test for excess
errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_1.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
    PASS: libgomp.fortran/rwlock_1.f90   -O3 -g  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_1.f90   -O3 -g  execution test
    PASS: libgomp.fortran/rwlock_1.f90   -Os  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_1.f90   -Os  execution test
    PASS: libgomp.fortran/rwlock_2.f90   -O0  (test for excess errors)
    PASS: libgomp.fortran/rwlock_2.f90   -O0  execution test
    PASS: libgomp.fortran/rwlock_2.f90   -O1  (test for excess errors)
    PASS: libgomp.fortran/rwlock_2.f90   -O1  execution test
    PASS: libgomp.fortran/rwlock_2.f90   -O2  (test for excess errors)
    PASS: libgomp.fortran/rwlock_2.f90   -O2  execution test
    PASS: libgomp.fortran/rwlock_2.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  (test for excess
errors)
    PASS: libgomp.fortran/rwlock_2.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
    PASS: libgomp.fortran/rwlock_2.f90   -O3 -g  (test for excess errors)
    PASS: libgomp.fortran/rwlock_2.f90   -O3 -g  execution test
    PASS: libgomp.fortran/rwlock_2.f90   -Os  (test for excess errors)
    PASS: libgomp.fortran/rwlock_2.f90   -Os  execution test
    PASS: libgomp.fortran/rwlock_3.f90   -O0  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_3.f90   -O0  execution test
    PASS: libgomp.fortran/rwlock_3.f90   -O1  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_3.f90   -O1  execution test
    PASS: libgomp.fortran/rwlock_3.f90   -O2  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_3.f90   -O2  execution test
    PASS: libgomp.fortran/rwlock_3.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  (test for excess
errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_3.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
    PASS: libgomp.fortran/rwlock_3.f90   -O3 -g  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_3.f90   -O3 -g  execution test
    PASS: libgomp.fortran/rwlock_3.f90   -Os  (test for excess errors)
    WARNING: program timed out.
    FAIL: libgomp.fortran/rwlock_3.f90   -Os  execution test

All-PASS on all x86_64-pc-linux-gnu systems that I've tested.

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[tschwinge at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

Thomas Schwinge <tschwinge at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|libfortran                  |testsuite
   Last reconfirmed|                            |2023-12-21
             Target|powerpc64le-linux-gnu       |
                 CC|                            |burnus at gcc dot gnu.org,
                   |                            |jakub at gcc dot gnu.org
             Status|UNCONFIRMED                 |NEW
     Ever confirmed|0                           |1

--- Comment #1 from Thomas Schwinge <tschwinge at gcc dot gnu.org> ---
Turns out, this isn't actually specific to powerpc64le-linux-gnu, but rather
the following: my testing where I saw the timeouts was not build-tree 'make
check' testing, but instead "installed" testing (where you invoke 'runtest' on
a 'make install'ed GCC tree).  In that case, r266482 "Tweak libgomp env vars in
parallel make check (take 2)" is not in effect, that is, there's no limiting to
'OMP_NUM_THREADS=8'.

For example, manually running the '-O0' variant of
'libgomp.fortran/rwlock_1.f90' on a "big-iron" x86_64-pc-linux-gnu system:

    $ grep ^model\ name < /proc/cpuinfo | uniq -c
        256 model name  : AMD EPYC 7V13 64-Core Processor
    $ \time env OMP_NUM_THREADS=[...] LD_LIBRARY_PATH=[...] ./rwlock_1.exe

..., I produce the following data on an idle system:

'OMP_NUM_THREADS=8':

    0.16user 0.56system 0:02.36elapsed 31%CPU (0avgtext+0avgdata
4452maxresident)k
    0.17user 0.54system 0:02.30elapsed 30%CPU (0avgtext+0avgdata
4532maxresident)k

'OMP_NUM_THREADS=16':

    0.40user 1.03system 0:04.52elapsed 31%CPU (0avgtext+0avgdata
5832maxresident)k
    0.49user 0.99system 0:04.39elapsed 33%CPU (0avgtext+0avgdata
5876maxresident)k

'OMP_NUM_THREADS=32':

    0.98user 2.36system 0:09.33elapsed 35%CPU (0avgtext+0avgdata
8528maxresident)k
    0.98user 2.25system 0:09.02elapsed 35%CPU (0avgtext+0avgdata
8548maxresident)k

'OMP_NUM_THREADS=64':

    1.82user 5.83system 0:18.44elapsed 41%CPU (0avgtext+0avgdata
13952maxresident)k
    1.54user 6.03system 0:18.22elapsed 41%CPU (0avgtext+0avgdata
13996maxresident)k

'OMP_NUM_THREADS=128':

    3.71user 12.41system 0:38.02elapsed 42%CPU (0avgtext+0avgdata
24376maxresident)k
    3.96user 12.52system 0:39.34elapsed 41%CPU (0avgtext+0avgdata
24476maxresident)k

'OMP_NUM_THREADS=256' (or not set, for that matter):

    9.65user 25.19system 1:20.93elapsed 43%CPU (0avgtext+0avgdata
45816maxresident)k
    8.99user 25.82system 1:19.40elapsed 43%CPU (0avgtext+0avgdata
45636maxresident)k

For comparison, if I remove 'LD_LIBRARY_PATH', such that the system-wide GCC 10
libraries are used, I get for the latter case:

    9.28user 24.54system 1:22.09elapsed 41%CPU (0avgtext+0avgdata
45588maxresident)k
    11.26user 24.51system 1:24.32elapsed 42%CPU (0avgtext+0avgdata
45712maxresident)k

..., so only a little bit of an improvement of the new "rwlock" libgfortran vs.
old "mutex" GCC 10 one, curiously.  (But supposedly that depends on the
hardware or other factors?)

Anyway: should these test cases be limiting themselves to some lower
'OMP_NUM_THREADS', for example via 'num_threads' clauses?

The powerpc64le-linux-gnu systems:

    $ grep ^cpu < /proc/cpuinfo | uniq -c

    160 cpu             : POWER8 (raw), altivec supported

    152 cpu             : POWER8NVL (raw), altivec supported

    128 cpu             : POWER9, altivec supported

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #2 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Yeah, perhaps just num_threads(8) to all of those?
Why is there
total_threads = omp_get_max_threads ()
btw, when nothing uses it?

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #3 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
BTW, even in non-installed testing there is no OMP_NUM_THREADS cap if one just
uses make check and not -jN.  Or when OMP_NUM_THREADS is set in the environment
to some value.

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[lipeng.zhu at intel dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

Lipeng Zhu <lipeng.zhu at intel dot com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lipeng.zhu at intel dot com

--- Comment #4 from Lipeng Zhu <lipeng.zhu at intel dot com> ---
(In reply to Jakub Jelinek from comment #2)
> Yeah, perhaps just num_threads(8) to all of those?
> Why is there
> total_threads = omp_get_max_threads ()
> btw, when nothing uses it?
Yes, this code is not used, we can delete it.

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #5 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
I can confirm that on x86_64-linux with 16 cores/32 threads even the -O0
rwlock_1 and rwlock_3 tests aren't that slow, byt with OMP_NUM_THREADS=1024
and higher rwlock_1 STOPs:
$ OMP_NUM_THREADS=256 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
2.21user 8.88system 0:00.88elapsed 1260%CPU (0avgtext+0avgdata
47324maxresident)k
0inputs+206848outputs (0major+29321minor)pagefaults 0swaps
$ OMP_NUM_THREADS=512 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
4.59user 14.41system 0:02.29elapsed 829%CPU (0avgtext+0avgdata
89464maxresident)k
0inputs+413696outputs (0major+55232minor)pagefaults 0swaps
$ OMP_NUM_THREADS=1024 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
STOP 2
STOP 2
Command exited with non-zero status 2
0.04user 0.78system 0:00.14elapsed 586%CPU (0avgtext+0avgdata
18976maxresident)k
0inputs+1672outputs (2major+4138minor)pagefaults 0swaps
$ OMP_NUM_THREADS=1024 LD_LIBRARY_PATH=../.libs/ time ./rwlock_3.exe
13.57user 49.00system 0:17.66elapsed 354%CPU (0avgtext+0avgdata
26588maxresident)k
0inputs+0outputs (1major+5987minor)pagefaults 0swaps
$ OMP_NUM_THREADS=2048 LD_LIBRARY_PATH=../.libs/ time ./rwlock_3.exe
38.15user 134.83system 0:51.26elapsed 337%CPU (0avgtext+0avgdata
45860maxresident)k
0inputs+0outputs (0major+10844minor)pagefaults 0swaps

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #6 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
But guess another case is on a loaded system.
When building on Fedora for distro builds on all of aarch64, powerpc64le, s390x
or x86_64 I see:
+FAIL: libgomp.fortran/rwlock_1.f90   -O0  execution test
+FAIL: libgomp.fortran/rwlock_1.f90   -O1  execution test
+FAIL: libgomp.fortran/rwlock_1.f90   -O2  execution test
+FAIL: libgomp.fortran/rwlock_1.f90   -O3 -fomit-frame-pointer -funroll-loops
-fpeel-loops -ftracer -finline-functions  execution test
+FAIL: libgomp.fortran/rwlock_1.f90   -O3 -g  execution test
+FAIL: libgomp.fortran/rwlock_1.f90   -Os  execution test
and in all those cases it was make -j{8,6,3,8} check.  Though, for number of
threads <= 8 we don't do the OMP_NUM_THREADS limiting and so it uses the
defaults.
Anyway, see e.g.
https://kojipkgs.fedoraproject.org//work/tasks/4298/110644298/build.log
and one can uudecode the make check logs from that file.

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #7 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
wget .../build.log; sed -n /^begin/,/^end/p build.log | uudecode

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #8 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Execution timeout is: 300
spawn [open ...]
STOP 2
Internal Error: Trying to free nonempty asynchronous unit

Error termination. Backtrace:
Internal Error: Trying to free nonempty asynchronous unit

Error termination. Backtrace:
#0  0x7f986429289e in free_async_unit
        at ../../../libgfortran/io/async.c:211
#1  0x7f9864288729 in close_unit_1
        at ../../../libgfortran/io/unit.c:759
#2  0x401644 in ???
#3  0x7f986432cd7d in ???
#4  0x7f9863e87c90 in ???
#5  0x7f9863f07fbb in ???
#6  0xffffffffffffffff in ???
FAIL: libgomp.fortran/rwlock_1.f90   -O0  execution test
is what I see in the log.  Or just STOP 2.

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[lipeng.zhu at intel dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #9 from Lipeng Zhu <lipeng.zhu at intel dot com> ---
Since I still can't reproduce the failure on my side :(, just curious, will the
new added 'rwlock' test cases failed on mutex lock?

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #10 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
For the distro builds the log files are all I have (all logs in
https://koji.fedoraproject.org/koji/taskinfo?taskID=110644223 )
For what I can reproduce on my box (rwlock_1.exe built in the
x86_64-pc-linux-gnu/libgomp/testsuite subdirectory using the -O0
compilation line from libgomp*/*.sep:
$ OMP_NUM_THREADS=4096 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
STOP 2
At line 28 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
2187, file = '***_tst.dat')
Fortran runtime error: End of file
STOP 2
Command exited with non-zero status 2
0.11user 1.19system 0:00.19elapsed 691%CPU (0avgtext+0avgdata
51824maxresident)k
0inputs+368outputs (0major+12303minor)pagefaults 0swaps
$ OMP_NUM_THREADS=4096 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
STOP 2
STOP 2
Command exited with non-zero status 2
0.09user 1.03system 0:00.17elapsed 650%CPU (0avgtext+0avgdata
49988maxresident)k
0inputs+288outputs (0major+11835minor)pagefaults 0swaps
$ OMP_NUM_THREADS=4096 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
STOP 2
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3798)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3151)
Internal Error: Unit number changed
Internal Error: Unit number changed

Error termination. Backtrace:

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 795)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 46)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
2386)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1841)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3692)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3880)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 947)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
2930)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 565)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 323)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1651)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1684)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3659)

Error termination. Backtrace:
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 846)
Fortran runtime error: Cannot open file '846_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
2224)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:

Could not print backtrace: /proc/self/exe: Too many open files
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3913)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
#0  0x7f9728be6a3e
#1  0x7f9728be7509
#2  0x7f9728be805f
#0  0x7f9728be6a3e
#3  0x7f9728e29128
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1183)
#1  0x7f9728be7509
#4  0x7f9728e2940d
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
#2  0x7f9728be805f
#5  0x40140f
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 860)

Error termination. Backtrace:
#3  0x7f9728e29128
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
2565)
#6  0x7f9728ab430d
Fortran runtime error: Cannot open file '860_tst.dat': Too many open files
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3722)
#4  0x7f9728e2940d
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
#7  0x7f97288fbe5c

Error termination. Backtrace:
#0  0x7f9728be6a3e
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
#5  0x40140f

Error termination. Backtrace:
#8  0x7f97289818ef
#1  0x7f9728be7509

Error termination. Backtrace:
#2  0x7f9728be805f
#6  0x7f9728ab430d
#9  0xffffffffffffffff
#3  0x7f9728e29128
#0  0x7f9728be6a3e
#7  0x7f97288fbe5c
#4  0x7f9728e2940d
#0  0x7f9728be6a3e
#1  0x7f9728be7509
#8  0x7f97289818ef
#5  0x40140f
#0  0x7f9728be6a3e
#1  0x7f9728be7509
#2  0x7f9728be805f
#9  0xffffffffffffffff
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 707)
#6  0x7f9728ab430d
#1  0x7f9728be7509
#2  0x7f9728be805f
#3  0x7f9728e29128
Fortran runtime error: Cannot open file '707_tst.dat': Too many open files
#7  0x7f97288fbe5c
#2  0x7f9728be805f
#3  0x7f9728e29128

Error termination. Backtrace:
#8  0x7f97289818ef
#3  0x7f9728e29128
#4  0x7f9728e2940d
#9  0xffffffffffffffff
#4  0x7f9728e2940d
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
3382)
#5  0x40140f
#5  0x40140f
#0  0x7f9728be6a3e
#6  0x7f9728ab430d
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1534)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
#6  0x7f9728ab430d
#1  0x7f9728be7509
#7  0x7f97288fbe5c
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
#7  0x7f97288fbe5c
#2  0x7f9728be805f
#8  0x7f97289818ef

Error termination. Backtrace:
#8  0x7f97289818ef
#3  0x7f9728e29128
#9  0xffffffffffffffff
#9  0xffffffffffffffff
#4  0x7f9728e2940d
#0  0x7f9728be6a3e
#5  0x40140f
#1  0x7f9728be7509
#6  0x7f9728ab430d
#0  0x7f9728be6a3e
#2  0x7f9728be805f
#7  0x7f97288fbe5c
#1  0x7f9728be7509
#3  0x7f9728e29128
#8  0x7f97289818ef
#2  0x7f9728be805f
#4  0x7f9728e2940d
#9  0xffffffffffffffff
#5  0x40140f
#3  0x7f9728e29128
#6  0x7f9728ab430d
#4  0x7f9728e2940d
#7  0x7f97288fbe5c
#5  0x40140f
#8  0x7f97289818ef
#6  0x7f9728ab430d
#9  0xffffffffffffffff
#7  0x7f97288fbe5c
#8  0x7f97289818ef
#9  0xffffffffffffffff
Command exited with non-zero status 2
0.14user 1.54system 0:00.21elapsed 783%CPU (0avgtext+0avgdata
59916maxresident)k
0inputs+184outputs (2major+14287minor)pagefaults 0swaps
$ OMP_NUM_THREADS=2048 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
At line 28 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1330, file = '***_tst.dat')
Fortran runtime error: End of file
STOP 2
At line 28 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1103, file = '***_tst.dat')
Fortran runtime error: End of file
STOP 2
Command exited with non-zero status 2
0.06user 0.63system 0:00.14elapsed 472%CPU (0avgtext+0avgdata
27116maxresident)k
0inputs+480outputs (0major+6134minor)pagefaults 0swaps
$ OMP_NUM_THREADS=2048 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
STOP 2
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1759)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1734)
Internal Error: Unit number changed
Internal Error: Unit number changed

Error termination. Backtrace:

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1011)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1702)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 898)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 903)
Internal Error: Unit number changed
Internal Error: Unit number changed

Error termination. Backtrace:

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 545)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1181)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 106)
Internal Error: Unit number changed
Internal Error: Unit number changed

Error termination. Backtrace:

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1622)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 243)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1677)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 25 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1321)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 69)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 842)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1285)
Fortran runtime error: Cannot open file '69_tst.dat': Too many open files
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1340)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1723)
Fortran runtime error: Cannot open file '842_tst.dat': Too many open files
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
At line 25 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 495)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 186)

Error termination. Backtrace:
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:

Error termination. Backtrace:
Fortran runtime error: Cannot open file '495_tst.dat': Too many open files
Fortran runtime error: Cannot open file '186_tst.dat': Too many open files

Error termination. Backtrace:

Error termination. Backtrace:

Error termination. Backtrace:

Error termination. Backtrace:

Could not print backtrace: /proc/self/exe: Too many open files

Could not print backtrace: /proc/self/exe: Too many open files
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1427)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1658)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1180)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
#0  0x7f5db0b78a3e
#1  0x7f5db0b79509
#2  0x7f5db0b7a05f
#3  0x7f5db0dbb128
#4  0x7f5db0dbb40d
#5  0x40140f
#6  0x7f5db0a4630d
#7  0x7f5db088de5c
#8  0x7f5db09138ef
#9  0xffffffffffffffff
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1073)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
#0  0x7f5db0b78a3e
#1  0x7f5db0b79509
#2  0x7f5db0b7a05f
#3  0x7f5db0dbb128
#4  0x7f5db0dbb40d
#5  0x40140f
#6  0x7f5db0a4630d
#7  0x7f5db088de5c
#8  0x7f5db09138ef
#9  0xffffffffffffffff
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1027)
Fortran runtime error: Cannot open file '***_tst.dat': Too many open files

Error termination. Backtrace:
#0  0x7f5db0b78a3e
#1  0x7f5db0b79509
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 952)
Fortran runtime error: Cannot open file '952_tst.dat': Too many open files

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 33)
Command exited with non-zero status 2
0.07user 0.85system 0:00.14elapsed 659%CPU (0avgtext+0avgdata
37472maxresident)k
0inputs+576outputs (0major+8565minor)pagefaults 0swaps
$ OMP_NUM_THREADS=1024 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
STOP 2
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1029)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 803)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 25 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 867)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 712)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 123)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 162)
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 35)
Internal Error: Unit number changed
Internal Error: Unit number changed

Error termination. Backtrace:

Error termination. Backtrace:
At line 25 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 374)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 25 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 543)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 676)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 404)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 913)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit = 985)
Internal Error: Unit number changed

Error termination. Backtrace:
#0  0x7efceee2ea12 in ???
#1  0x7efceee2f509 in ???
#2  0x7efceee2fc38 in ???
#3  0x7efcef071196 in ???
#4  0x7efcef07140d in ???
#5  0x40140f in MAIN__._omp_fn.0
        at
/home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90:17
#6  0x7efceecfc30d in gomp_thread_start
        at ../../../libgomp/team.c:129
#7  0x7efceeb43e5c in ???
#8  0x7efceebc98ef in ???
#9  0xffffffffffffffff in ???
Command exited with non-zero status 3
0.12user 0.77system 0:00.14elapsed 608%CPU (0avgtext+0avgdata
36608maxresident)k
192inputs+4368outputs (1major+8568minor)pagefaults 0swaps

With OMP_NUM_THREADS=512 it works.

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[lipeng.zhu at intel dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #11 from Lipeng Zhu <lipeng.zhu at intel dot com> ---
(In reply to Jakub Jelinek from comment #10)
>For what I can reproduce on my box (rwlock_1.exe built in the
>x86_64-pc-linux-gnu/libgomp/testsuite subdirectory using the -O0
>compilation line from libgomp*/*.sep:
>$ OMP_NUM_THREADS=4096 LD_LIBRARY_PATH=../.libs/ time ./rwlock_1.exe
>
> Error termination. Backtrace:
> At line 17 of file
> /home/jakub/src/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
> 1180)
> Fortran runtime error: Cannot open file '***_tst.dat': Too many open files
> 
> 
> With OMP_NUM_THREADS=512 it works.

I can reproduce the "Too many open files" error on my local.
The error is caused by the limited open files setting. 
$ ulimit -n
In existing rwlock test case, it will maintain a fd for each thread, with
OMP_NUM_THREADS growing, it reached the system limitation. 

>Anyway: should these test cases be limiting themselves to some lower
> 'OMP_NUM_THREADS', for example via 'num_threads' clauses?

This proposal could avoid this kind of issue.

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[lipeng.zhu at intel dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #12 from Lipeng Zhu <lipeng.zhu at intel dot com> ---
(In reply to Lipeng Zhu from comment #9)
> Since I still can't reproduce the failure on my side :(, just curious, will
> the new added 'rwlock' test cases failed on mutex lock?

OK, I rerun these rwlock* test cases on mutex lock, it reported the similar
error:

$ export OMP_NUM_THREADS=1024 && time ./rwlock_1.exe
At line 28 of file
/home/lipzhu/codes/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
1017, file = '***_tst.dat')
Fortran runtime error: End of file
STOP 2
At line 17 of file
/home/lipzhu/codes/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
587)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/lipzhu/codes/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
732)
Internal Error: Unit number changed

Error termination. Backtrace:
At line 17 of file
/home/lipzhu/codes/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
963)
At line 17 of file
/home/lipzhu/codes/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
980)
Internal Error: Unit number changed
Internal Error: Unit number changed

Error termination. Backtrace:

Error termination. Backtrace:
At line 17 of file
/home/lipzhu/codes/gcc/libgomp/testsuite/libgomp.fortran/rwlock_1.f90 (unit =
421)
Internal Error: Unit number changed

Error termination. Backtrace:
#0  0x7ff587881223 in already_open
#0  0x7ff587881223 in already_open
#0  0x7ff587881223 in already_open
        at ../../../libgfortran/io/open.c:742
        at ../../../libgfortran/io/open.c:742
        at ../../../libgfortran/io/open.c:742
#1  0x400def in ???
#1  0x400def in ???
#1  0x400def in ???
#0  0x7ff587881223 in already_open
        at ../../../libgfortran/io/open.c:742
#1  0x400def in ???
#0  0x7ff587881223 in already_open
        at ../../../libgfortran/io/open.c:742
#1  0x400def in ???
#2  0x7ff586e21d2d in gomp_thread_start
        at ../../../libgomp/team.c:129
#3  0x7ff5866081c9 in ???
#2  0x7ff586e21d2d in gomp_thread_start
        at ../../../libgomp/team.c:129
#3  0x7ff5866081c9 in ???
#4  0x7ff586239e72 in ???
#4  0x7ff586239e72 in ???
#5  0xffffffffffffffff in ???
#5  0xffffffffffffffff in ???
#2  0x7ff586e21d2d in gomp_thread_start
        at ../../../libgomp/team.c:129
#3  0x7ff5866081c9 in ???
#4  0x7ff586239e72 in ???
#2  0x7ff586e21d2d in gomp_thread_start
#5  0xffffffffffffffff in ???
        at ../../../libgomp/team.c:129
#3  0x7ff5866081c9 in ???
#4  0x7ff586239e72 in ???
#5  0xffffffffffffffff in ???
#2  0x7ff586e21d2d in gomp_thread_start
        at ../../../libgomp/team.c:129
#3  0x7ff5866081c9 in ???
#4  0x7ff586239e72 in ???
#5  0xffffffffffffffff in ???

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[lipeng.zhu at intel dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #13 from Lipeng Zhu <lipeng.zhu at intel dot com> ---
OK, I think I find the root cause of this error, when thread number greater
than 1000, the file_name = 1000_tst.dat, character(11) will overflow. This will
generate the same file_name like ***_tst.dat. 

diff --git a/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
b/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
index f90ecbeb00f..1c271ae031d 100644
--- a/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
+++ b/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
@@ -7,13 +7,12 @@ program main
   use omp_lib
   implicit none
   integer:: unit_number, v1, v2, i
-  character(11) :: file_name
+  character(16) :: file_name
   character(3) :: async = "no"
   !$omp parallel private (unit_number, v1, v2, file_name, async, i)
     do i = 0, 100
       unit_number = 10 + omp_get_thread_num ()
-      write (file_name, "(I3, A)") unit_number, "_tst.dat"
-      file_name = adjustl(file_name)
+      write (file_name, "(I5.5, A)") unit_number, "_tst.dat"
       open (unit_number, file=file_name, asynchronous="yes")
       ! call inquire with file parameter to test find_file in unix.c
       inquire (file=file_name, asynchronous=async)

[Bug testsuite/113005] 'libgomp.fortran/rwlock_1.f90', 'libgomp.fortran/rwlock_3.f90' execution test timeouts

[lipeng.zhu at intel dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113005

--- Comment #14 from Lipeng Zhu <lipeng.zhu at intel dot com> ---
(In reply to Lipeng Zhu from comment #13)
> OK, I think I find the root cause of this error, when thread number greater
> than 1000, the file_name = 1000_tst.dat, character(11) will overflow. This
> will generate the same file_name like ***_tst.dat. 
> 
> diff --git a/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
> b/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
> index f90ecbeb00f..1c271ae031d 100644
> --- a/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
> +++ b/libgomp/testsuite/libgomp.fortran/rwlock_1.f90
> @@ -7,13 +7,12 @@ program main
>    use omp_lib
>    implicit none
>    integer:: unit_number, v1, v2, i
> -  character(11) :: file_name
> +  character(16) :: file_name
>    character(3) :: async = "no"
>    !$omp parallel private (unit_number, v1, v2, file_name, async, i)
>      do i = 0, 100
>        unit_number = 10 + omp_get_thread_num ()
> -      write (file_name, "(I3, A)") unit_number, "_tst.dat"
> -      file_name = adjustl(file_name)
> +      write (file_name, "(I5.5, A)") unit_number, "_tst.dat"
>        open (unit_number, file=file_name, asynchronous="yes")
>        ! call inquire with file parameter to test find_file in unix.c
>        inquire (file=file_name, asynchronous=async)


Hi Thomas,

Can you help to verify if this draft patch will fix the error on your side?