Running the compat testsuite in "non-mirror" mode

Running the compat testsuite in "non-mirror" mode

[Eric Botcazou]

Hello Janis,

I'd like to automatically run the compat testsuite in "non-mirror" mode on the 
SPARC, to assert internal consistency for vector calling conventions, in 
addition to the regular "mirror" mode.

What is the best approach to achieve that?  Can I duplicate compat.exp and 
override ALT_CC_UNDER_TEST and COMPAT_OPTIONS in a sparc-compat.exp driver?  
Would it be feasible to invoke compat.exp from another driver, for example 
located in the gcc.target/sparc directory?

Thanks in advance.

-- 
Eric Botcazou

Re: Running the compat testsuite in "non-mirror" mode

[Janis Johnson]

On Thu, Nov 18, 2004 at 08:16:06AM +0100, Eric Botcazou wrote:
> Hello Janis,
> 
> I'd like to automatically run the compat testsuite in "non-mirror" mode on the 
> SPARC, to assert internal consistency for vector calling conventions, in 
> addition to the regular "mirror" mode.
> 
> What is the best approach to achieve that?  Can I duplicate compat.exp and 
> override ALT_CC_UNDER_TEST and COMPAT_OPTIONS in a sparc-compat.exp driver?  
> Would it be feasible to invoke compat.exp from another driver, for example 
> located in the gcc.target/sparc directory?

I'm not sure what you mean by "mirror" and "non-mirror" modes.

When ALT_CC_UNDER_TEST (or ALT_CXX_UNDER_TEST) is defined we build
the following object files:

  x_tst.o and y_tst.o are compiled using CC_UNDER TEST
  x_alt.o and y_alt.o are compiled using ALT_CC_UNDER TEST
 
Executables are built from each pair:

  x_tst.o and y_tst.o
  x_tst.o and y_alt.o
  x_alt.o and y_tst.o
  x_alt.o and y_alt.o

Everything in the first executable is compiled by the compiler under
test; is that what you want?  Testing an executable for which everything
is compiled by the alternate compiler makes it easier to detect failures
in the mixed executables that are due to the alternate compiler.

Janis

Re: Running the compat testsuite in "non-mirror" mode

[Eric Botcazou]

> I'm not sure what you mean by "mirror" and "non-mirror" modes.

Automatically running the compat testsuite with the newly built compiler and a 
pair of (different) options.

-- 
Eric Botcazou

Re: Running the compat testsuite in "non-mirror" mode

[Janis Johnson]

On Fri, Nov 19, 2004 at 12:24:48AM +0100, Eric Botcazou wrote:
> > I'm not sure what you mean by "mirror" and "non-mirror" modes.
> 
> Automatically running the compat testsuite with the newly built compiler and a 
> pair of (different) options.

Like the lists of pairs of options in COMPAT_OPTIONS that are described
in the compat test documentation in sourcebuild.texi?

Janis

Re: Running the compat testsuite in "non-mirror" mode

[Eric Botcazou]

> Like the lists of pairs of options in COMPAT_OPTIONS that are described
> in the compat test documentation in sourcebuild.texi?

Yes, but automatically.  In other words, I'd like to automatically run the 
compat testsuite twice on the SPARC, in mirror mode and in non-mirror mode 
with a hardcoded pair of options.

-- 
Eric Botcazou

Re: Running the compat testsuite in "non-mirror" mode

[Janis Johnson]

On Fri, Nov 19, 2004 at 01:09:32AM +0100, Eric Botcazou wrote:
> > Like the lists of pairs of options in COMPAT_OPTIONS that are described
> > in the compat test documentation in sourcebuild.texi?
> 
> Yes, but automatically.  In other words, I'd like to automatically run the 
> compat testsuite twice on the SPARC, in mirror mode and in non-mirror mode 
> with a hardcoded pair of options.

I'm sorry, I still don't understand what you mean my mirror and non-mirror
modes.

Janis

Re: Running the compat testsuite in "non-mirror" mode

[Eric Botcazou]

> I'm sorry, I still don't understand what you mean my mirror and non-mirror
> modes.

OK, sorry for being obtuse.

The compat testsuite is currently automatically run only once, in what I call 
mirror mode, that is the newly built compiler is tested against (an identical 
copy of) itself.  While this is useful for catching problems in the 
argument/return value handling machinery of the back-end, this is only 
moderately useful for compatibility purposes.  Of course this is 
customizable, but only externally (i.e. manually or with an external Makefile 
for example).

What I would like to have is the possibility to automatically run (i.e. with a 
bare make -k check-gcc) the compat testsuite twice, once in the mirror mode 
described above and once in a non-mirror mode.  The latter would mean that 
the newly built compiler is tested against a slight variation of itself, i.e. 
typically with a non-default option, that could have an impact on the calling 
conventions, turned on.  So it would be possible to specify in the compat 
testsuite harness (e.g. a platform-specific driver) that it should be 
automatically run twice on a particular platform, in mirror mode and with an 
hardcoded pair (-mfoo/-mno-foo) of options.

-- 
Eric Botcazou

Re: Running the compat testsuite in "non-mirror" mode

[Janis Johnson]

On Fri, Nov 19, 2004 at 09:17:04AM +0100, Eric Botcazou wrote:
> > I'm sorry, I still don't understand what you mean my mirror and non-mirror
> > modes.
> 
> OK, sorry for being obtuse.
> 
> The compat testsuite is currently automatically run only once, in what I call 
> mirror mode, that is the newly built compiler is tested against (an identical 
> copy of) itself.  While this is useful for catching problems in the 
> argument/return value handling machinery of the back-end, this is only 
> moderately useful for compatibility purposes.  Of course this is 
> customizable, but only externally (i.e. manually or with an external Makefile 
> for example).
> 
> What I would like to have is the possibility to automatically run (i.e. with a 
> bare make -k check-gcc) the compat testsuite twice, once in the mirror mode 
> described above and once in a non-mirror mode.  The latter would mean that 
> the newly built compiler is tested against a slight variation of itself, i.e. 
> typically with a non-default option, that could have an impact on the calling 
> conventions, turned on.  So it would be possible to specify in the compat 
> testsuite harness (e.g. a platform-specific driver) that it should be 
> automatically run twice on a particular platform, in mirror mode and with an 
> hardcoded pair (-mfoo/-mno-foo) of options.

I think I understand now.  For anyone testing a particular target, the
compat tests would be run first using the compiler under test to compile
both parts of each test using default options, and again using different
sets of options for the two parts of each test to make sure those
options don't affect binary compatibility.  Is that right?

This is probably easy to set up in the compat.exp file for each compat
testsuite; I'll take a look.

Janis

Re: Running the compat testsuite in "non-mirror" mode

[Eric Botcazou]

> I think I understand now.  For anyone testing a particular target, the
> compat tests would be run first using the compiler under test to compile
> both parts of each test using default options, and again using different
> sets of options for the two parts of each test to make sure those
> options don't affect binary compatibility.  Is that right?

Yes; a maintainer of a given platform could decide that it is important to 
always test a particular couple of options on this platform, and set the 
appropriate bits in the right driver file.

> This is probably easy to set up in the compat.exp file for each compat
> testsuite; I'll take a look.

Thanks.  And thanks for your patience too. :-)

-- 
Eric Botcazou

Re: Running the compat testsuite in "non-mirror" mode

[Janis Johnson]

On Fri, Nov 19, 2004 at 07:12:59PM +0100, Eric Botcazou wrote:
> > I think I understand now.  For anyone testing a particular target, the
> > compat tests would be run first using the compiler under test to compile
> > both parts of each test using default options, and again using different
> > sets of options for the two parts of each test to make sure those
> > options don't affect binary compatibility.  Is that right?
> 
> Yes; a maintainer of a given platform could decide that it is important to 
> always test a particular couple of options on this platform, and set the 
> appropriate bits in the right driver file.
> 
> > This is probably easy to set up in the compat.exp file for each compat
> > testsuite; I'll take a look.

Does this do the kind of thing you want?  This is just an example, not
options I actually want to add for powerpc.

Other files that set COMPAT_OPTIONS are lib/compat.exp,
gcc.dg/compat/struct-layout-1.exp, and g++.dg/compat/compat.exp.  This
continues to allow the default sets of options to be overridden.

Janis

Index: gcc.dg/compat/compat.exp
===================================================================
RCS file: /opt/gcc-cvs/gcc/gcc/testsuite/gcc.dg/compat/compat.exp,v
retrieving revision 1.1
diff -u -p -r1.1 compat.exp
--- gcc.dg/compat/compat.exp	5 May 2003 21:59:35 -0000	1.1
+++ gcc.dg/compat/compat.exp	20 Nov 2004 00:25:50 -0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2002, 2003 Free Software Foundation, Inc.
+# Copyright (C) 2002, 2003, 2004 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -34,6 +34,20 @@ global GCC_UNDER_TEST
 # Load procedures from common libraries. 
 load_lib standard.exp
 load_lib gcc.exp
+load_lib target-supports.exp
+
+# Some targets run the tests multiple times with different sets of options.
+if ![info exists COMPAT_OPTIONS] {
+    if { [istarget powerpc*-*-linux*] && [is-effective-target vmx_hw] } {
+	set COMPAT_OPTIONS [list \
+		[list {} {}] \
+		[list {-maltivec -mabi=altivec} {-mno-altivec -mabi=altivec}]]
+    } elseif [istarget powerpc*-*-linux*] {
+	set COMPAT_OPTIONS [list \
+		[list {} {}] \
+		[list {-O1} {-O2}]]
+    }
+}
 
 #
 # compat-use-alt-compiler -- make the alternate compiler the default
@@ -65,7 +79,8 @@ proc compat-use-tst-compiler { } {
 }
 
 # Load the language-independent compabibility support procedures.
-# This must be done after the compat-use-*-compiler definitions.
+# This must be done after the compat-use-*-compiler definitions and
+# after defining COMPAT_OPTIONS.
 load_lib compat.exp
 
 gcc_init

Re: Running the compat testsuite in "non-mirror" mode

[Eric Botcazou]

> Does this do the kind of thing you want?  This is just an example, not
> options I actually want to add for powerpc.

Looks perfect.  Let me give it a whirl tomorrow on SPARC though.

> Other files that set COMPAT_OPTIONS are lib/compat.exp,
> gcc.dg/compat/struct-layout-1.exp, and g++.dg/compat/compat.exp.  This
> continues to allow the default sets of options to be overridden.

I think this is the right granularity.  I'd like to be able to run the C 
compat testsuite twice, without necessarily doing the same for the 
struct-layout-1 testsuite.

-- 
Eric Botcazou

Re: Running the compat testsuite in "non-mirror" mode

[Eric Botcazou]

> Does this do the kind of thing you want?  This is just an example, not
> options I actually want to add for powerpc.

This worked fine on the SPARC:

Index: gcc.dg/compat/compat.exp
===================================================================
RCS file: /cvs/gcc/gcc/gcc/testsuite/gcc.dg/compat/compat.exp,v
retrieving revision 1.1
diff -u -r1.1 compat.exp
--- gcc.dg/compat/compat.exp	5 May 2003 21:59:35 -0000	1.1
+++ gcc.dg/compat/compat.exp	21 Nov 2004 20:18:28 -0000
@@ -35,6 +35,15 @@
 load_lib standard.exp
 load_lib gcc.exp
 
+# Some targets run the tests multiple times with different sets of options.
+if ![info exists COMPAT_OPTIONS] {
+    if { [istarget sparc*-*-*] } {
+	set COMPAT_OPTIONS [list \
+	    [list {} {}] \
+	    [list {-mcpu=ultrasparc -mvis} {}]]
+    }
+}
+
 #
 # compat-use-alt-compiler -- make the alternate compiler the default
 # 

-- 
Eric Botcazou