[PATCH 0/3] Unbork testsuite for multlib setups

[PATCH 0/3] Unbork testsuite for multlib setups

[Vineet Gupta]

Hi,

This fixes the current broken multilib testing on trunk.
1/3 is th actual fix
2/3 is preventive and
3/3 is debug aid in case this bites someone else in future.

Thx,
-Vineet

Kito Cheng (1):
  RISC-V: Add missing torture-init and torture-finish for rvv.exp

Vineet Gupta (2):
  testsuite: Unbork multilib testing on RISC-V (and any target really)
  testsuite: print any leaking torture options for debugging

 gcc/testsuite/gcc.misc-tests/i386-prefetch.exp | 14 +++++++-------
 gcc/testsuite/gcc.target/riscv/rvv/rvv.exp     |  3 +++
 gcc/testsuite/lib/torture-options.exp          |  6 +++---
 3 files changed, 13 insertions(+), 10 deletions(-)

-- 
2.34.1

[PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Vineet Gupta]

Multilib testing on trunk is currently busted (and surprisingly this
affects any/all targets but it seems nobody cares). We currently get the
following splat:

| ERROR: tcl error code NONE
| ERROR: torture-init: torture_without_loops is not empty as expected

And this takes down pretty much all of testsuite.

|               ========= Summary of gcc testsuite =========
|                            | # of unexpected case / # of unique unexpected case
|                            |          gcc |          g++ |     gfortran |
| rv64imafdc/  lp64d/ medlow | 5421 /     4 |    1 /     1 |   72 /    12 |
| rv32imafdc/ ilp32d/ medlow | 5422 /     5 |    3 /     2 |   72 /    12 |
|   rv32imac/  ilp32/ medlow |  391 /     5 |    3 /     2 |  109 /    19 |
|   rv64imac/   lp64/ medlow | 5422 /     5 |    1 /     1 |  109 /    19 |

There have been recent improvements in test harness around pairing of
torture-{init,finish} and checking for leaking torture options. This
however triggers a latent bug introduced way back in 2009: commit 3dd1415dc88
"i386-prefetch.exp: Skip tests when multilib flags contain -march" which
missed a pairing torture-finish. It was benign so far but in the new
regime it causes extra state "torture-init-done" confusing the 2nd round of
tests (in multilib).

This fix moves the early exit outside of torture-{init,finish} bracket
and brings RISC-V testing back to sanity.

| rv64imafdc/  lp64d/ medlow |    3 /     2 |    1 /     1 |   72 /    12 |
| rv32imafdc/ ilp32d/ medlow |    4 /     3 |    3 /     2 |   72 /    12 |
|   rv32imac/  ilp32/ medlow |    3 /     2 |    3 /     2 |  109 /    19 |
|   rv64imac/   lp64/ medlow |    5 /     4 |    1 /     1 |  109 /    19 |

gcc/testsuite:
	* gcc.misc-tests/i386-prefetch.exp: Move early return outside
	  the torture-{init,finish}

Signed-off-by: Vineet Gupta <vineetg@rivosinc.com>
---
 gcc/testsuite/gcc.misc-tests/i386-prefetch.exp | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp b/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp
index ad9e56a54bcf..7101b1e30576 100644
--- a/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp
+++ b/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp
@@ -82,6 +82,13 @@ if $tracelevel then {
     strace $tracelevel
 }
 
+if { [board_info target exists multilib_flags]
+     && [string match "* -march=*" " [board_info target multilib_flags] "] } {
+    # Multilib flags come after the -march flags we pass and override
+    # them, so skip these tests when such flags are passed.
+    return
+}
+
 # Load support procs.
 load_lib gcc-dg.exp
 load_lib torture-options.exp
@@ -90,13 +97,6 @@ load_lib torture-options.exp
 dg-init
 torture-init
 
-if { [board_info target exists multilib_flags]
-     && [string match "* -march=*" " [board_info target multilib_flags] "] } {
-    # Multilib flags come after the -march flags we pass and override
-    # them, so skip these tests when such flags are passed.
-    return
-}
-
 set-torture-options $PREFETCH_NONE
 gcc-dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/i386-pf-none-*.c]] "" ""
 
-- 
2.34.1

[PATCH 2/3] RISC-V: Add missing torture-init and torture-finish for rvv.exp

[Vineet Gupta]

From: Kito Cheng <kito.cheng@sifive.com>

This is in line with recent test harness expectations and is a
preventive change as it doesn't actually fix any errors.

gcc/testsuite/ChangeLog:

	* gcc.target/riscv/rvv/rvv.exp: Add torture-init and
	torture-finish.

Signed-off-by: Vineet Gupta <vineetg@rivosinc.com>
---
 gcc/testsuite/gcc.target/riscv/rvv/rvv.exp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp b/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp
index 5e69235a268c..7ab7456d1d15 100644
--- a/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp
+++ b/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp
@@ -39,6 +39,7 @@ if [istarget riscv32-*-*] then {
 
 # Initialize `dg'.
 dg-init
+torture-init
 
 # Main loop.
 set CFLAGS "$DEFAULT_CFLAGS -march=$gcc_march -mabi=$gcc_mabi -O3"
@@ -90,5 +91,7 @@ foreach op $AUTOVEC_TEST_OPTS {
 dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/autovec/vls-vlmax/*.\[cS\]]] \
 	"-std=c99 -O3 -ftree-vectorize --param riscv-autovec-preference=fixed-vlmax" $CFLAGS
 
+torture-finish
+
 # All done.
 dg-finish
-- 
2.34.1

[PATCH 3/3] testsuite: print any leaking torture options for debugging

[Vineet Gupta]

This was helpful when debugging the recent multilib testsuite failure.

gcc/testsuite:
	* lib/torture-options.exp: print the value of non-empty options:
	torture_without_loops, torture_with_loops, LTO_TORTURE_OPTIONS.

Signed-off-by: Vineet Gupta <vineetg@rivosinc.com>
---
 gcc/testsuite/lib/torture-options.exp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gcc/testsuite/lib/torture-options.exp b/gcc/testsuite/lib/torture-options.exp
index d00d07e9378d..dfb536d1d96c 100644
--- a/gcc/testsuite/lib/torture-options.exp
+++ b/gcc/testsuite/lib/torture-options.exp
@@ -23,15 +23,15 @@ proc torture-init { args } {
     global torture_without_loops global_with_loops
 
     if [info exists torture_without_loops] {
-	error "torture-init: torture_without_loops is not empty as expected"
+	error "torture-init: torture_without_loops is not empty as expected = \"${torture_without_loops}\""
     }
     if [info exists torture_with_loops] {
-	error "torture-init: torture_with_loops is not empty as expected"
+	error "torture-init: torture_with_loops is not empty as expected = \"${torture_with_loops}\""
     }
 
     global LTO_TORTURE_OPTIONS
     if [info exists LTO_TORTURE_OPTIONS] {
-	error "torture-init: LTO_TORTURE_OPTIONS is not empty as expected"
+	error "torture-init: LTO_TORTURE_OPTIONS is not empty as expected =  \"${LTO_TORTURE_OPTIONS}\""
     }
     set LTO_TORTURE_OPTIONS ""
     if [check_effective_target_lto] {
-- 
2.34.1

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Jeff Law]

On 5/31/23 10:25, Vineet Gupta wrote:
> Multilib testing on trunk is currently busted (and surprisingly this
> affects any/all targets but it seems nobody cares). We currently get the
> following splat:
I wouldn't say that nobody cares, it just hasn't bubbled up on anyone's 
priority list yet (most developers aren't working on targets that make 
heavy use of multilibs).

But probably more importantly, this problem seems to not be triggering 
on all multilib targets.  For example, I just examined my tester's build 
logs and couldn't see this on the H8/300 or V850 ports.  Which begs the 
question, why?

Jeff

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Iain Sandoe]

> On 31 May 2023, at 18:57, Jeff Law via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
> 
> 
> 
> On 5/31/23 10:25, Vineet Gupta wrote:
>> Multilib testing on trunk is currently busted (and surprisingly this
>> affects any/all targets but it seems nobody cares). We currently get the
>> following splat:
> I wouldn't say that nobody cares, it just hasn't bubbled up on anyone's priority list yet (most developers aren't working on targets that make heavy use of multilibs).
> 
> But probably more importantly, this problem seems to not be triggering on all multilib targets.  For example, I just examined my tester's build logs and couldn't see this on the H8/300 or V850 ports.  Which begs the question, why?

I do have a multilib problem [with libgomp] on Darwin (which has been noticed : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109951) but it is not obvious how the fix proposed would solve this - unless it’s some subtle change in global content for the multilib options.

(testing anyway)

thanks
Iain

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Vineet Gupta]

On 5/31/23 10:57, Jeff Law wrote:
>
>
> On 5/31/23 10:25, Vineet Gupta wrote:
>> Multilib testing on trunk is currently busted (and surprisingly this
>> affects any/all targets but it seems nobody cares). We currently get the
>> following splat:
> I wouldn't say that nobody cares, it just hasn't bubbled up on 
> anyone's priority list yet (most developers aren't working on targets 
> that make heavy use of multilibs).

Pardon my theatrics :-)

> But probably more importantly, this problem seems to not be triggering 
> on all multilib targets.  For example, I just examined my tester's 
> build logs and couldn't see this on the H8/300 or V850 ports.  Which 
> begs the question, why?

Are just in case, this is not running a subset using some stray 
RUNTESTFLAGS.

Yes I'm curious to see why others are not seeing it. Could you rerun 
upstream with following debug (and avoid -j when running the testsuite 
just to serialize the logs - the problem does happen for -j runs too 
though). Then in the logs we could see if init/finish get out of sync 
around the culprit file (or my case at least)


--->
diff --git a/gcc/testsuite/lib/torture-options.exp 
b/gcc/testsuite/lib/torture-options.exp
index dfb536d1d96c..95a6f818fded 100644
--- a/gcc/testsuite/lib/torture-options.exp
+++ b/gcc/testsuite/lib/torture-options.exp
@@ -22,6 +22,8 @@
  proc torture-init { args } {
      global torture_without_loops global_with_loops

+    send_user "\n\n===  torture-init\n"
+
      if [info exists torture_without_loops] {
         error "torture-init: torture_without_loops is not empty as 
expected = \"${torture_without_loops}\""
      }
@@ -116,6 +118,8 @@ proc set-torture-options { args } {
  proc torture-finish { args } {
      global torture_without_loops torture_with_loops

+    send_user "\n\n===  torture-finish\n"
+
      if [info exists torture_without_loops] {
         unset torture_without_loops
      } else {

--->8---

FWIW I'd like to be able to test stuff cross-arch too (at least x86, 
aarch64 and a few others).

Thx,
-Vineet

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Vineet Gupta]

On 5/31/23 11:13, Iain Sandoe wrote:
> I do have a multilib problem [with libgomp] on Darwin (which has been noticed :https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109951) but it is not obvious how the fix proposed would solve this - unless it’s some subtle change in global content for the multilib options.

For the issue I was seeing, the actual multilib content didn't matter. 
Its just that the test needed to be run for more than 1 rounds so that 
the imbalanced torture-init from 1st multlib/round created the state 
which triggered errors for 2nd round...


Schedule of variations:
     riscv-sim/-march=rv32imac/-mabi=ilp32/-mcmodel=medlow
     riscv-sim/-march=rv32imafdc/-mabi=ilp32d/-mcmodel=medlow
     riscv-sim/-march=rv64imac/-mabi=lp64/-mcmodel=medlow
     riscv-sim/-march=rv64imafdc/-mabi=lp64d/-mcmodel=medlow

Running target riscv-sim/-march=rv32imac/-mabi=ilp32/-mcmodel=medlow
Using 
/scratch/vineetg/gnu/INSTALL/tc-up-230524-273895500425/share/dejagnu/baseboards/riscv-sim.exp 
as board description file for target.
Using 
/scratch/vineetg/gnu/INSTALL/tc-up-230524-273895500425/share/dejagnu/config/sim.exp 
as generic interface file for target.
Using 
/scratch/vineetg/gnu/INSTALL/tc-up-230524-273895500425/share/dejagnu/baseboards/basic-sim.exp 
as board description file for target.
Using 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/config/default.exp 
as tool-and-target-specific interface file.
Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.c-torture/compile/compile.exp 
...
===  torture-init
===  torture-finish

Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.c-torture/execute/builtins/builtins.exp 
...
===  torture-init
===  torture-finish

Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.c-torture/execute/execute.exp 
...
===  torture-init
===  torture-finish
....
Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp 
...
===  torture-init
             ^^^^^^
Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.misc-tests/linkage.exp 
...
Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.misc-tests/matrix1.exp 
...
...
Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.target/xtensa/xtensa.exp 
...
Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.test-framework/test-framework.exp 
...
skipping test framework tests, CHECK_TEST_FRAMEWORK is not defined

         === gcc Summary for 
riscv-sim/-march=rv32imac/-mabi=ilp32/-mcmodel=medlow ===

# of expected passes        136964
# of unexpected failures    4
# of unexpected successes    3
# of expected failures        1072
# of unsupported tests        3052

Running target riscv-sim/-march=rv32imafdc/-mabi=ilp32d/-mcmodel=medlow 
<--- 2nd round
Using 
/scratch/vineetg/gnu/INSTALL/tc-up-230524-273895500425/share/dejagnu/baseboards/riscv-sim.exp 
as board description file for target.
Using 
/scratch/vineetg/gnu/INSTALL/tc-up-230524-273895500425/share/dejagnu/config/sim.exp 
as generic interface file for target.
Using 
/scratch/vineetg/gnu/INSTALL/tc-up-230524-273895500425/share/dejagnu/baseboards/basic-sim.exp 
as board description file for target.
Using 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/config/default.exp 
as tool-and-target-specific interface file.
Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.c-torture/compile/compile.exp 
...

--- gcc-dg-runtest : NOT calling torture-init
--- gcc-dg-runtest : NOT calling torture-finish

Running 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.c-torture/execute/builtins/builtins.exp 
...

===  torture-init
ERROR: tcl error sourcing 
/scratch/vineetg/gnu/toolchain-upstream-pristine/gcc/gcc/testsuite/gcc.c-torture/execute/builtins/builtins.exp.
ERROR: tcl error code NONE
ERROR: torture-init: torture_without_loops is not empty as expected = "{ 
-O0 } { -O1 } { -O2 } { -O3 -g } { -Os } { -O2 -flto 
-fno-use-linker-plugin -flto-partition=none } { -O2 -flto 
-fuse-linker-plugin -fno-fat-lto-objects }"
     while executing

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Thomas Schwinge]

Hi!

First, Vineet, great that you've now tracked this down!  :-) Indeed
"early exit" vs. 'torture-finish' was exactly the issue that I suspected.

It may not be what you originally intended, but I hope at least you've
learned some things about DejaGnu/TCL...  ;-P

Yesterday, I actually had begun looking into this.  To avoid the big
download and having to wait for a lot of packages to be build with your
'riscv-gnu-toolchain' recipe:
<https://inbox.sourceware.org/44506218-70bd-0b7b-a560-744bb24376a9@rivosinc.com>,
I intended to do just a quick GCC build on compile farm gcc92, which
(a) didn't turn out to be quick, and (b) eventually failed due to
<https://gcc.gnu.org/PR106271>
"Bootstrap on RISC-V on Ubuntu 22.04 LTS: bits/libc-header-start.h: No such file or directory"...

(I'm now running 'riscv-gnu-toolchain' to verify this, and another thing.)

Before we push your patch, let me please verify that it indeed doesn't
change any 'gcc.misc-tests/i386-prefetch.exp' semantics, and:

On 2023-05-31T19:13:01+0100, Iain Sandoe via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
>> On 31 May 2023, at 18:57, Jeff Law via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
>> On 5/31/23 10:25, Vineet Gupta wrote:
>>> Multilib testing on trunk is currently busted (and surprisingly this
>>> affects any/all targets but it seems nobody cares). We currently get the
>>> following splat:
>> I wouldn't say that nobody cares, it just hasn't bubbled up on anyone's priority list yet (most developers aren't working on targets that make heavy use of multilibs).

So I regularely do build x86_64 GNU/Linux with default '-m64' plus '-m32'
multilib -- but of course, there's no "early exit" for those, as there's
no 'string match "* -march=*" " [board_info target multilib_flags] "'...

>> But probably more importantly, this problem seems to not be triggering on all multilib targets.  For example, I just examined my tester's build logs and couldn't see this on the H8/300 or V850 ports.  Which begs the question, why?

..., which may be the case for those, too?  In other words: the problem
only shows up if '-march=[...]' appears in the flags, which indeed may
not be a common thing?  I'll cross-verify this with x86_64 and
'-march=[...]' flags.

And, I still intend to figure out why this issue apparently disappears
with my recent 'LTO_TORTURE_OPTIONS' patches reverted:
<https://inbox.sourceware.org/ad8a98da-0231-7a95-017b-ea5d8ae65678@rivosinc.com>.


Otherwise:

> I do have a multilib problem [with libgomp] on Darwin (which has been noticed : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109951) but it is not obvious how the fix proposed would solve this - unless it’s some subtle change in global content for the multilib options.
>
> (testing anyway)

No, this is really a separate issue.  I understand what's happening, and
have an idea about how to address this that I'll post later.


Grüße
 Thomas
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Thomas Schwinge]

Hi!

On 2023-06-01T09:24:20+0200, I wrote:
> First, Vineet, great that you've now tracked this down!  :-) Indeed
> "early exit" vs. 'torture-finish' was exactly the issue that I suspected.
>
> It may not be what you originally intended, but I hope at least you've
> learned some things about DejaGnu/TCL...  ;-P
>
> Yesterday, I actually had begun looking into this.  To avoid the big
> download and having to wait for a lot of packages to be build with your
> 'riscv-gnu-toolchain' recipe:
> <https://inbox.sourceware.org/44506218-70bd-0b7b-a560-744bb24376a9@rivosinc.com>,
> I intended to do just a quick GCC build on compile farm gcc92, which
> (a) didn't turn out to be quick, and (b) eventually failed due to
> <https://gcc.gnu.org/PR106271>
> "Bootstrap on RISC-V on Ubuntu 22.04 LTS: bits/libc-header-start.h: No such file or directory"...
>
> (I'm now running 'riscv-gnu-toolchain' to verify this, and another thing.)

If running that with just 'RUNTESTFLAGS="i386-prefetch.exp"', I get:

    [...]
    Schedule of variations:
        riscv-sim/-march=rv32imac/-mabi=ilp32/-mcmodel=medlow
        riscv-sim/-march=rv32imafdc/-mabi=ilp32d/-mcmodel=medlow
        riscv-sim/-march=rv64imac/-mabi=lp64/-mcmodel=medlow
        riscv-sim/-march=rv64imafdc/-mabi=lp64d/-mcmodel=medlow

    Running target riscv-sim/-march=rv32imac/-mabi=ilp32/-mcmodel=medlow
    [...]
    Running [...]/gcc.misc-tests/i386-prefetch.exp ...

                    === gcc Summary for riscv-sim/-march=rv32imac/-mabi=ilp32/-mcmodel=medlow ===

    Running target riscv-sim/-march=rv32imafdc/-mabi=ilp32d/-mcmodel=medlow
    [...]
    Running [...]/gcc.misc-tests/i386-prefetch.exp ...
    ERROR: tcl error sourcing [...]/gcc.misc-tests/i386-prefetch.exp.
    ERROR: tcl error code NONE
    ERROR: torture-init: LTO_TORTURE_OPTIONS is not empty as expected
    [...]

..., which indeed complains about 'LTO_TORTURE_OPTIONS' (which relates to
my recent changes in that area, which I now do understand, see below).

The issue is that indeed 'torture-init' now does set
'LTO_TORTURE_OPTIONS', whereas 'torture_without_loops',
'torture_with_loops' are set only when 'set-torture-options' is called.

> Before we push your patch, let me please verify that it indeed doesn't
> change any 'gcc.misc-tests/i386-prefetch.exp' semantics

Done.

> and:
>
> On 2023-05-31T19:13:01+0100, Iain Sandoe via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
>>> On 31 May 2023, at 18:57, Jeff Law via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
>>> On 5/31/23 10:25, Vineet Gupta wrote:
>>>> Multilib testing on trunk is currently busted (and surprisingly this
>>>> affects any/all targets but it seems nobody cares). We currently get the
>>>> following splat:
>>> I wouldn't say that nobody cares, it just hasn't bubbled up on anyone's priority list yet (most developers aren't working on targets that make heavy use of multilibs).
>
> So I regularely do build x86_64 GNU/Linux with default '-m64' plus '-m32'
> multilib -- but of course, there's no "early exit" for those, as there's
> no 'string match "* -march=*" " [board_info target multilib_flags] "'...
>
>>> But probably more importantly, this problem seems to not be triggering on all multilib targets.  For example, I just examined my tester's build logs and couldn't see this on the H8/300 or V850 ports.  Which begs the question, why?
>
> ..., which may be the case for those, too?  In other words: the problem
> only shows up if '-march=[...]' appears in the flags, which indeed may
> not be a common thing?  I'll cross-verify this with x86_64 and
> '-march=[...]' flags.

That is the crucial thing indeed.  Vineet, please note that in the Git
commit log.  That is, instead of "Multilib testing", say "Multilib
testing involving '-march=[...]' flags", or similar.

The ERRORs do reproduce with x86_64 GNU/Linux with:

    RUNTESTFLAGS='--target_board=unix\{-m32,-m64,-march=generic,-march=generic\} i386-prefetch.exp'

..., for example.  Here, '-m32' behaves as expected, '-m64' behaves as
expected, the first '-march=generic' does the 'torture-init' and "early
exit", the second '-march=generic' then again does 'torture-init' and
runs into the error condition.

> And, I still intend to figure out why this issue apparently disappears
> with my recent 'LTO_TORTURE_OPTIONS' patches reverted:
> <https://inbox.sourceware.org/ad8a98da-0231-7a95-017b-ea5d8ae65678@rivosinc.com>.

In the "old world", 'torture-init', *not* followed by
'set-torture-options', *not* followed by 'torture-finish', then another
'torture-init' was not a problem -- but in the "new world" it now is.

This also explains my confusion; the original report was:

    ERROR: torture-init: torture_without_loops is not empty as expected

..., note: not 'LTO_TORTURE_OPTIONS' but 'torture_without_loops', and
those I'd not directly touched in my recent changes, which had made me
confused.

The 'torture_without_loops' error condition now does arise if there's a
'torture-init', *not* followed by 'set-torture-options', *not* followed
by 'torture-finish' (that is, 'i386-prefetch.exp'), then 'gcc-dg-runtest'
('riscv.exp', for example), which internally skips 'torture-init' (due to
'torture-init-done', due to 'LTO_TORTURE_OPTIONS'), but it does
'set-torture-options', then skips 'torture-finish', and then any next
'torture-init' detects the mismatch, thus ERRORs.


I can be convinced otherwise, but I still maintain my position that
requiring/checking proper bracketing of 'torture-init', 'torture-finish'
is advisable, and therefore propose to not re-do my 'LTO_TORTURE_OPTIONS'
changes, but indeed suggest to apply Vineet's patch, with a minor change,
see below.  (I cannot formally approve it, however; testsuite maintainers
CCed.)


As for your proposed patch:
<https://inbox.sourceware.org/20230531162534.119952-2-vineetg@rivosinc.com>,
I suggest to move the "early exit" in front of all the setup code, that
is, right after license header:

    [...]
    # <http://www.gnu.org/licenses/>.

    [HERE]

    # Test that the correct data prefetch instructions (SSE or 3DNow! variant,
    [...]


Grüße
 Thomas


> Otherwise:
>
>> I do have a multilib problem [with libgomp] on Darwin (which has been noticed : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109951) but it is not obvious how the fix proposed would solve this - unless it’s some subtle change in global content for the multilib options.
>>
>> (testing anyway)
>
> No, this is really a separate issue.  I understand what's happening, and
> have an idea about how to address this that I'll post later.
>
>
> Grüße
>  Thomas
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955

Re: [PATCH 2/3] RISC-V: Add missing torture-init and torture-finish for rvv.exp

[Thomas Schwinge]

Hi!

On 2023-05-31T09:25:33-0700, Vineet Gupta <vineetg@rivosinc.com> wrote:
> From: Kito Cheng <kito.cheng@sifive.com>
>
> This is in line with recent test harness expectations and is a
> preventive change as it doesn't actually fix any errors.
>
> gcc/testsuite/ChangeLog:
>
>       * gcc.target/riscv/rvv/rvv.exp: Add torture-init and
>       torture-finish.
>
> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com>
> ---
>  gcc/testsuite/gcc.target/riscv/rvv/rvv.exp | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp b/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp
> index 5e69235a268c..7ab7456d1d15 100644
> --- a/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp
> +++ b/gcc/testsuite/gcc.target/riscv/rvv/rvv.exp
> @@ -39,6 +39,7 @@ if [istarget riscv32-*-*] then {
>
>  # Initialize `dg'.
>  dg-init
> +torture-init
>
>  # Main loop.
>  set CFLAGS "$DEFAULT_CFLAGS -march=$gcc_march -mabi=$gcc_mabi -O3"
> @@ -90,5 +91,7 @@ foreach op $AUTOVEC_TEST_OPTS {
>  dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/autovec/vls-vlmax/*.\[cS\]]] \
>       "-std=c99 -O3 -ftree-vectorize --param riscv-autovec-preference=fixed-vlmax" $CFLAGS
>
> +torture-finish
> +
>  # All done.
>  dg-finish

I suggest to drop this patch: 'gcc.target/riscv/rvv/rvv.exp' isn't doing
anything with torture testing flags etc., but (in addition to
'dg-runtest') just calls 'gcc-dg-runtest', which internally does
'torture-init', 'torture-finish' -- like in a number of other '*.exp'
files.  As you say, this patch "doesn't actually fix any errors".


Grüße
 Thomas
-----------------
Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Jeff Law]

On 6/1/23 01:24, Thomas Schwinge wrote:

> 
>>> But probably more importantly, this problem seems to not be triggering on all multilib targets.  For example, I just examined my tester's build logs and couldn't see this on the H8/300 or V850 ports.  Which begs the question, why?
> 
> ..., which may be the case for those, too?  In other words: the problem
> only shows up if '-march=[...]' appears in the flags, which indeed may
> not be a common thing?  I'll cross-verify this with x86_64 and
> '-march=[...]' flags.
Correct.  Those do not use -march.  They have distinct flags for turning 
on sub-variants.  So for example on the H8 -ms turns on H8/S code 
generation, -msx turns on H8/S, etc.  V850 has different options, but 
follows the same basic principle.


Jeff

Re: [PATCH 3/3] testsuite: print any leaking torture options for debugging

[Jeff Law]

On 5/31/23 10:25, Vineet Gupta wrote:
> This was helpful when debugging the recent multilib testsuite failure.
> 
> gcc/testsuite:
> 	* lib/torture-options.exp: print the value of non-empty options:
> 	torture_without_loops, torture_with_loops, LTO_TORTURE_OPTIONS.
OK
jeff

Re: [PATCH 1/3] testsuite: Unbork multilib testing on RISC-V (and any target really)

[Jeff Law]

On 5/31/23 10:25, Vineet Gupta wrote:
> Multilib testing on trunk is currently busted (and surprisingly this
> affects any/all targets but it seems nobody cares). We currently get the
> following splat:
> 
> | ERROR: tcl error code NONE
> | ERROR: torture-init: torture_without_loops is not empty as expected
> 
> And this takes down pretty much all of testsuite.
> 
> |               ========= Summary of gcc testsuite =========
> |                            | # of unexpected case / # of unique unexpected case
> |                            |          gcc |          g++ |     gfortran |
> | rv64imafdc/  lp64d/ medlow | 5421 /     4 |    1 /     1 |   72 /    12 |
> | rv32imafdc/ ilp32d/ medlow | 5422 /     5 |    3 /     2 |   72 /    12 |
> |   rv32imac/  ilp32/ medlow |  391 /     5 |    3 /     2 |  109 /    19 |
> |   rv64imac/   lp64/ medlow | 5422 /     5 |    1 /     1 |  109 /    19 |
> 
> There have been recent improvements in test harness around pairing of
> torture-{init,finish} and checking for leaking torture options. This
> however triggers a latent bug introduced way back in 2009: commit 3dd1415dc88
> "i386-prefetch.exp: Skip tests when multilib flags contain -march" which
> missed a pairing torture-finish. It was benign so far but in the new
> regime it causes extra state "torture-init-done" confusing the 2nd round of
> tests (in multilib).
> 
> This fix moves the early exit outside of torture-{init,finish} bracket
> and brings RISC-V testing back to sanity.
> 
> | rv64imafdc/  lp64d/ medlow |    3 /     2 |    1 /     1 |   72 /    12 |
> | rv32imafdc/ ilp32d/ medlow |    4 /     3 |    3 /     2 |   72 /    12 |
> |   rv32imac/  ilp32/ medlow |    3 /     2 |    3 /     2 |  109 /    19 |
> |   rv64imac/   lp64/ medlow |    5 /     4 |    1 /     1 |  109 /    19 |
> 
> gcc/testsuite:
> 	* gcc.misc-tests/i386-prefetch.exp: Move early return outside
> 	  the torture-{init,finish}
OK after addressing Thomas's comments which I think just amounted to 
moving the code to a different place and adjusting the comments in the 
commit message.

jeff

Re: [PATCH 2/3] RISC-V: Add missing torture-init and torture-finish for rvv.exp

[Jeff Law]

On 5/31/23 10:25, Vineet Gupta wrote:
> From: Kito Cheng <kito.cheng@sifive.com>
> 
> This is in line with recent test harness expectations and is a
> preventive change as it doesn't actually fix any errors.
> 
> gcc/testsuite/ChangeLog:
> 
> 	* gcc.target/riscv/rvv/rvv.exp: Add torture-init and
> 	torture-finish.
Thomas's recommendation was to drop this as it doesn't change any 
observed behavior.  Do you agree with that recommendation?

jeff

Re: [PATCH 2/3] RISC-V: Add missing torture-init and torture-finish for rvv.exp

[Vineet Gupta]

On 6/1/23 07:54, Jeff Law wrote:
>
>
> On 5/31/23 10:25, Vineet Gupta wrote:
>> From: Kito Cheng <kito.cheng@sifive.com>
>>
>> This is in line with recent test harness expectations and is a
>> preventive change as it doesn't actually fix any errors.
>>
>> gcc/testsuite/ChangeLog:
>>
>>     * gcc.target/riscv/rvv/rvv.exp: Add torture-init and
>>     torture-finish.
> Thomas's recommendation was to drop this as it doesn't change any 
> observed behavior.  Do you agree with that recommendation?

Yep dropped now.

Thx,
-Vineet

[Committed] testsuite: Unbork multilib setups using -march flags (RISC-V)

[Vineet Gupta]

RISC-V multilib testing is currently busted with follow splat all over:

|    Schedule of variations:
|        riscv-sim/-march=rv64imafdc/-mabi=lp64d/-mcmodel=medlow
|        riscv-sim/-march=rv32imafdc/-mabi=ilp32d/-mcmodel=medlow
|        riscv-sim/-march=rv32imac/-mabi=ilp32/-mcmodel=medlow
|        riscv-sim/-march=rv64imac/-mabi=lp64/-mcmodel=medlow
...
...
| ERROR: tcl error code NONE
| ERROR: torture-init: torture_without_loops is not empty as expected

causing insane amount of false failures.

|               ========= Summary of gcc testsuite =========
|                            | # of unexpected case / # of unique unexpected case
|                            |          gcc |          g++ |     gfortran |
| rv64imafdc/  lp64d/ medlow | 5421 /     4 |    1 /     1 |    6 /     1 |
| rv32imafdc/ ilp32d/ medlow | 5422 /     5 |    3 /     2 |    6 /     1 |
|   rv32imac/  ilp32/ medlow |  391 /     5 |    3 /     2 |   43 /     8 |
|   rv64imac/   lp64/ medlow | 5422 /     5 |    1 /     1 |   43 /     8 |

The error splat itself is from recent test harness improvements for stricter
checks for torture-{init,finish} pairing. But the real issue is a latent bug
from 2009: commit 3dd1415dc88, ("i386-prefetch.exp: Skip tests when multilib
flags contain -march") which added an "early exit" condition to i386-prefetch.exp
which could potentially cause an unpaired torture-{init,finish}.

The early exit only happens in a multlib setup using -march in flags
which is what RISC-V happens to use, hence the reason this was only seen
on RISC-V multilib testing.

Moving the early exit outside of torture-{init,finish} bracket
reinstates RISC-V testing.

| rv64imafdc/  lp64d/ medlow |    3 /     2 |    1 /     1 |    6 /     1 |
| rv32imafdc/ ilp32d/ medlow |    4 /     3 |    3 /     2 |    6 /     1 |
|   rv32imac/  ilp32/ medlow |    3 /     2 |    3 /     2 |   43 /     8 |
|   rv64imac/   lp64/ medlow |    5 /     4 |    1 /     1 |   43 /     8 |

gcc/testsuite:
	* gcc.misc-tests/i386-prefetch.exp: Move early return outside
	the torture-{init,finish}

Signed-off-by: Vineet Gupta <vineetg@rivosinc.com>
---
 gcc/testsuite/gcc.misc-tests/i386-prefetch.exp | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp b/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp
index ad9e56a54bcf..98aab506cba0 100644
--- a/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp
+++ b/gcc/testsuite/gcc.misc-tests/i386-prefetch.exp
@@ -14,6 +14,13 @@
 # along with GCC; see the file COPYING3.  If not see
 # <http://www.gnu.org/licenses/>.
 
+if { [board_info target exists multilib_flags]
+     && [string match "* -march=*" " [board_info target multilib_flags] "] } {
+    # Multilib flags come after the -march flags we pass and override
+    # them, so skip these tests when such flags are passed.
+    return
+}
+
 # Test that the correct data prefetch instructions (SSE or 3DNow! variant,
 # or none) are used for various i386 cpu-type and instruction set
 # extension options for __builtin_prefetch.  When using -mtune, specify
@@ -90,13 +97,6 @@ load_lib torture-options.exp
 dg-init
 torture-init
 
-if { [board_info target exists multilib_flags]
-     && [string match "* -march=*" " [board_info target multilib_flags] "] } {
-    # Multilib flags come after the -march flags we pass and override
-    # them, so skip these tests when such flags are passed.
-    return
-}
-
 set-torture-options $PREFETCH_NONE
 gcc-dg-runtest [lsort [glob -nocomplain $srcdir/$subdir/i386-pf-none-*.c]] "" ""
 
-- 
2.34.1

[Committed] testsuite: print any leaking torture options for debugging

[Vineet Gupta]

This was helpful when debugging the recent multilib testsuite failure.

gcc/testsuite:
	* lib/torture-options.exp: print the value of non-empty options:
	torture_without_loops, torture_with_loops, LTO_TORTURE_OPTIONS.

Signed-off-by: Vineet Gupta <vineetg@rivosinc.com>
---
 gcc/testsuite/lib/torture-options.exp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/gcc/testsuite/lib/torture-options.exp b/gcc/testsuite/lib/torture-options.exp
index d00d07e9378d..dfb536d1d96c 100644
--- a/gcc/testsuite/lib/torture-options.exp
+++ b/gcc/testsuite/lib/torture-options.exp
@@ -23,15 +23,15 @@ proc torture-init { args } {
     global torture_without_loops global_with_loops
 
     if [info exists torture_without_loops] {
-	error "torture-init: torture_without_loops is not empty as expected"
+	error "torture-init: torture_without_loops is not empty as expected = \"${torture_without_loops}\""
     }
     if [info exists torture_with_loops] {
-	error "torture-init: torture_with_loops is not empty as expected"
+	error "torture-init: torture_with_loops is not empty as expected = \"${torture_with_loops}\""
     }
 
     global LTO_TORTURE_OPTIONS
     if [info exists LTO_TORTURE_OPTIONS] {
-	error "torture-init: LTO_TORTURE_OPTIONS is not empty as expected"
+	error "torture-init: LTO_TORTURE_OPTIONS is not empty as expected =  \"${LTO_TORTURE_OPTIONS}\""
     }
     set LTO_TORTURE_OPTIONS ""
     if [check_effective_target_lto] {
-- 
2.34.1