[Bug testsuite/111969] New: RISC-V rv32gcv: 12 grouped flaky failures

[Bug testsuite/111969] New: RISC-V rv32gcv: 12 grouped flaky failures

[patrick at rivosinc dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111969

            Bug ID: 111969
           Summary: RISC-V rv32gcv: 12 grouped flaky failures
           Product: gcc
           Version: 14.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: testsuite
          Assignee: unassigned at gcc dot gnu.org
          Reporter: patrick at rivosinc dot com
  Target Milestone: ---

These 12 testcases are flaky on _only_ the rv32gcv glibc target. They either
all fail or all pass as a group.

FAIL: gcc.dg/nextafter-2.c execution test
FAIL: gfortran.dg/default_format_2.f90   -O0  execution test
FAIL: gfortran.dg/default_format_2.f90   -O1  execution test
FAIL: gfortran.dg/default_format_2.f90   -O2  execution test
FAIL: gfortran.dg/default_format_2.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/default_format_2.f90   -O3 -g  execution test
FAIL: gfortran.dg/default_format_2.f90   -Os  execution test
FAIL: gfortran.dg/default_format_denormal_2.f90   -O0  execution test
FAIL: gfortran.dg/default_format_denormal_2.f90   -O1  execution test
FAIL: gfortran.dg/default_format_denormal_2.f90   -O2  execution test
FAIL: gfortran.dg/default_format_denormal_2.f90   -O3 -fomit-frame-pointer
-funroll-loops -fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/default_format_denormal_2.f90   -O3 -g  execution test
FAIL: gfortran.dg/default_format_denormal_2.f90   -Os  execution test
FAIL: gfortran.dg/ieee/dec_math_1.f90   -O0  execution test
FAIL: gfortran.dg/ieee/dec_math_1.f90   -O1  execution test
FAIL: gfortran.dg/ieee/dec_math_1.f90   -O2  execution test
FAIL: gfortran.dg/ieee/dec_math_1.f90   -O3 -fomit-frame-pointer -funroll-loops
-fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/ieee/dec_math_1.f90   -O3 -g  execution test
FAIL: gfortran.dg/ieee/dec_math_1.f90   -Os  execution test
FAIL: gfortran.dg/ieee/large_1.f90   -O0  execution test
FAIL: gfortran.dg/ieee/large_2.f90   -O0  execution test
FAIL: gfortran.dg/ieee/large_2.f90   -O1  execution test
FAIL: gfortran.dg/ieee/large_2.f90   -O2  execution test
FAIL: gfortran.dg/ieee/large_2.f90   -O3 -fomit-frame-pointer -funroll-loops
-fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.dg/ieee/large_2.f90   -O3 -g  execution test
FAIL: gfortran.dg/ieee/large_2.f90   -Os  execution test
FAIL: gfortran.dg/intrinsic_pack_6.f90   -O2  execution test
FAIL: gfortran.dg/intrinsic_pack_6.f90   -O3 -g  execution test
FAIL: gfortran.dg/large_real_kind_2.F90   -O0  execution test
FAIL: gfortran.dg/round_4.f90   -O0  execution test
FAIL: gfortran.dg/zero_sized_3.f90   -O3 -fomit-frame-pointer -funroll-loops
-fpeel-loops -ftracer -finline-functions  execution test
FAIL: gfortran.fortran-torture/execute/intrinsic_count.f90 execution,  -O2
-fomit-frame-pointer -finline-functions -funroll-loops
FAIL: gfortran.fortran-torture/execute/intrinsic_matmul.f90 execution,  -O2
FAIL: gfortran.fortran-torture/execute/intrinsic_matmul.f90 execution,  -O2
-fbounds-check
FAIL: gfortran.fortran-torture/execute/intrinsic_matmul.f90 execution,  -O2
-fomit-frame-pointer -finline-functions
FAIL: gfortran.fortran-torture/execute/intrinsic_matmul.f90 execution,  -O2
-fomit-frame-pointer -finline-functions -funroll-loops
FAIL: gfortran.fortran-torture/execute/intrinsic_matmul.f90 execution,  -O3 -g
FAIL: gfortran.fortran-torture/execute/intrinsic_sum.f90 execution,  -O2
-fomit-frame-pointer -finline-functions -funroll-loops

r14-4098-gbdb7d85dde5 is the first time the CI observed these failures. It's
also relatively close to when we bumped the CI to QEMU v8.1.0 (~9 days after).

It can clearly be seen on the risc-v CI dashboard:
https://patrick-rivos.github.io/gcc-postcommit-ci/gcv where the fortran
failures for the rv32gcv target goes from ~23 -> ~15 failures and back again.

I'm investigating locally but not making much headway. Thought I'd chronicle
what I learn in a bugzilla in case anyone else has insight into what this might
be.

Marking this as a testsuite framework issue as that's my best guess currently.
I'll keep this issue updated as I investigate more.

[Bug testsuite/111969] RISC-V rv32gcv: 12 grouped flaky failures

[pinskia at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111969

Andrew Pinski <pinskia at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |testsuite-fail

--- Comment #1 from Andrew Pinski <pinskia at gcc dot gnu.org> ---
They all look fp related testcases which gives off simulator issue since you
said sometimes they work and sometimes they don't.

[Bug testsuite/111969] RISC-V rv32gcv: 12 grouped flaky failures

[schwab@linux-m68k.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111969

--- Comment #2 from Andreas Schwab <schwab@linux-m68k.org> ---
They never fail on real hardware (hifive unleashed).

[Bug testsuite/111969] RISC-V rv32gcv: 12 grouped flaky failures

[patrick at rivosinc dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111969

--- Comment #3 from Patrick O'Neill <patrick at rivosinc dot com> ---
These may be real failures that manifest in QEMU depending on the GCC used to
compile it.

I built qemu with r14-4098-gbdb7d85dde5 which caused the failures.
> QEMU_CPU=rv32,vlen=128,v=true,vext_spec=v1.0,Zve32f=true,Zve64f=true ../build-3/bin/qemu-riscv32 large_2.exe
>   0.333333333333333333333333333333333317         2.24271998593667819112500193394291495E+1644
> STOP 1

Building again with r14-4875-g9cf2e7441ee, I can execute both the new 
> QEMU_CPU=rv32,vlen=128,v=true,vext_spec=v1.0,Zve32f=true,Zve64f=true ./bin/qemu-riscv32 ./large_2.exe
_and_ the old exe:
> QEMU_CPU=rv32,vlen=128,v=true,vext_spec=v1.0,Zve32f=true,Zve64f=true ./bin/qemu-riscv32 ../../tc-oct-24-baseline-flaky/build/large_2.exe
with no issues/failures.

When I run r14-4875-g9cf2e7441ee large_2.exe on r14-4098-gbdb7d85dde5 QEMU, I
get the same failure:
> QEMU_CPU=rv32,vlen=128,v=true,vext_spec=v1.0,Zve32f=true,Zve64f=true ../build-3/bin/qemu-riscv32 ../../tc-oct-24-baseline-flaky/build/large_2.exe
>   0.333333333333333333333333333333333317         2.24271998593667819112500193394291495E+1644
> STOP 1

So it seems to be a QEMU issue that is caused by GCC?

[Bug testsuite/111969] RISC-V rv32gcv: 12 grouped flaky failures

[patrick at rivosinc dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111969

--- Comment #4 from Patrick O'Neill <patrick at rivosinc dot com> ---
Thinking about it more this doesn't really make sense since AFAIK QEMU is
compiled with the host compiler and shouldn't be affected by the GCC version
that compiles large_2.exe.

[Bug testsuite/111969] RISC-V rv32gcv: 12 grouped flaky failures

[patrick at rivosinc dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111969

--- Comment #5 from Patrick O'Neill <patrick at rivosinc dot com> ---
More info:
QEMU v8.1.0
glibc 2.37

Only seen on rv32gcv

Fails locally/CI:
r14-4098-gbdb7d85dde5

Pass locally/CI:
r14-4098-gbdb7d85dde5

Fails on CI, passes locally (?):
r14-4876-g7b2984ad76c

[Bug testsuite/111969] RISC-V rv32gcv: 12 grouped flaky failures

[patrick at rivosinc dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111969

--- Comment #6 from Patrick O'Neill <patrick at rivosinc dot com> ---
Mixed up my hashes when copy/pasting.
r14-4875-g9cf2e7441ee passes locally/CI