glue file built with compiler in PATH in out of tree testing

glue file built with compiler in PATH in out of tree testing

[Thomas Preudhomme]

Hi,

TL;DR: where to tell dejagnu about the compiler to use for building testglue?

== context ==

I've just found out that testglue.c is built using the compiler in PATH when 
doing out of tree testing rather than using the one specified by GCC_UNDER_TEST 
(or other *_UNDER_TEST). This is because testglue is built using build_wrapper 
in gcc/testsuite/lib/wrapper.exp which ultimately calls default_target_compile 
most of the time. The compiler is then looked for in the options (something 
starting with ^compiler), then in CC_FOR_TARGET (or similar *_FOR_TARGET) and 
then default to use the one in the PATH.

== problem ==

I'm not sure where is the proper place to fix this. Obviously setting 
CC_FOR_TARGET in contrib/test_installed or when calling runtest manually would 
work but I wonder if this would not be better fixed somewhere else? The rest of 
the testsuite seems happy with using GCC_UNDER_TEST, so surely testglue should 
be using the same thing? Should this be done in wrapper.exp? If yes, should we 
set *_FOR_TARGET? I don't see a way to pass ^compiler to build_wrapper's $flag 
variable.

Best regards,

Thomas

Re: glue file built with compiler in PATH in out of tree testing

[Joseph Myers]

On Wed, 6 Dec 2017, Thomas Preudhomme wrote:

> == problem ==
> 
> I'm not sure where is the proper place to fix this. Obviously setting
> CC_FOR_TARGET in contrib/test_installed or when calling runtest manually would
> work but I wonder if this would not be better fixed somewhere else? The rest
> of the testsuite seems happy with using GCC_UNDER_TEST, so surely testglue
> should be using the same thing? Should this be done in wrapper.exp? If yes,
> should we set *_FOR_TARGET? I don't see a way to pass ^compiler to
> build_wrapper's $flag variable.

One possibility is something along the lines of

set_board_info compiler "[find_gcc]"

or

set_board_info compiler $CC
set_board_info c++compiler $CXX

in your board file (and site.exp setting CC, CXX etc.).  I believe the 
thus-specified compiler will also be used for e.g. GDB testing.

(I'd also expect the compiler being tested to be in the PATH when doing 
installed testing.)

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: glue file built with compiler in PATH in out of tree testing

[Thomas Preudhomme]

Hi Joseph,

On 06/12/17 17:53, Joseph Myers wrote:
> On Wed, 6 Dec 2017, Thomas Preudhomme wrote:
> 
>> == problem ==
>>
>> I'm not sure where is the proper place to fix this. Obviously setting
>> CC_FOR_TARGET in contrib/test_installed or when calling runtest manually would
>> work but I wonder if this would not be better fixed somewhere else? The rest
>> of the testsuite seems happy with using GCC_UNDER_TEST, so surely testglue
>> should be using the same thing? Should this be done in wrapper.exp? If yes,
>> should we set *_FOR_TARGET? I don't see a way to pass ^compiler to
>> build_wrapper's $flag variable.
> 
> One possibility is something along the lines of
> 
> set_board_info compiler "[find_gcc]"
> 
> or
> 
> set_board_info compiler $CC
> set_board_info c++compiler $CXX
> 
> in your board file (and site.exp setting CC, CXX etc.).  I believe the
> thus-specified compiler will also be used for e.g. GDB testing.
> 
> (I'd also expect the compiler being tested to be in the PATH when doing
> installed testing.)

That seems to go counter to the --prefix option of contrib/test_installed which 
is meant to test a compiler at an arbitrary path. This suggest the compiler is 
not expected to be in PATH or in any dejagnu configuration file. Seems most of 
the testsuite seems to work fine with just GCC_UNDER_TEST, why not make it 
compile testglue using the value of that variable as well?

Best regards,

Thomas

Re: glue file built with compiler in PATH in out of tree testing

[Joseph Myers]

On Thu, 7 Dec 2017, Thomas Preudhomme wrote:

> That seems to go counter to the --prefix option of contrib/test_installed
> which is meant to test a compiler at an arbitrary path. This suggest the
> compiler is not expected to be in PATH or in any dejagnu configuration file.
> Seems most of the testsuite seems to work fine with just GCC_UNDER_TEST, why
> not make it compile testglue using the value of that variable as well?

My comments were on the general principle of installed testing, not 
contrib/test_installed specifically.

That is: you have a toolchain installed and want to run some DejaGnu tests 
with it.  Those tests are not necessarily tests of GCC; they might include 
GDB or newlib tests, for example.  Your board file needs to describe how 
to compile for the board, which is exactly what "set_board_info compiler" 
is for (and sometimes also "set_board_info cflags", "set_board_info 
ldflags", "set_board_info ldscript", especially if you have e.g. a 
bare-metal compiler configuration that only generates useful programs when 
a linker script for a particular board is specified - or if e.g. you're 
testing a GNU/Linux toolchain with glibc not installed in the target 
system's root directory, you need a board file that sets ldflags to use 
appropriate -Wl,-dynamic-linker -Wl,-rpath options).

So a generically useful board file describes the "compiler for the board".  
The compiler for the board may or may not be the same as the compiler 
being tested, if the testsuite you're running is a testsuite for a 
compiler at all.  (And if you're running the GCC compat tests you have 
*three* compilers involved - the compiler for the board, the GCC being 
tested and the other compiler you're testing ABI compatibility with.)

The argument for compiling testglue with the "compiler for the board" is 
simply that testglue is not part of the GCC testsuite at all; it's part of 
DejaGnu and needs to be built even in cases where you're running a 
non-compiler testsuite.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: glue file built with compiler in PATH in out of tree testing

[Thomas Preudhomme]

On 07/12/17 15:17, Joseph Myers wrote:
> On Thu, 7 Dec 2017, Thomas Preudhomme wrote:
> 
>> That seems to go counter to the --prefix option of contrib/test_installed
>> which is meant to test a compiler at an arbitrary path. This suggest the
>> compiler is not expected to be in PATH or in any dejagnu configuration file.
>> Seems most of the testsuite seems to work fine with just GCC_UNDER_TEST, why
>> not make it compile testglue using the value of that variable as well?
> 
> My comments were on the general principle of installed testing, not
> contrib/test_installed specifically.

Fair enough. However in the specific context of the GCC testsuite we might want 
the tested compiler to be the one used rather than the generic one for this board.

> 
> That is: you have a toolchain installed and want to run some DejaGnu tests
> with it.  Those tests are not necessarily tests of GCC; they might include
> GDB or newlib tests, for example.  Your board file needs to describe how
> to compile for the board, which is exactly what "set_board_info compiler"
> is for (and sometimes also "set_board_info cflags", "set_board_info
> ldflags", "set_board_info ldscript", especially if you have e.g. a
> bare-metal compiler configuration that only generates useful programs when
> a linker script for a particular board is specified - or if e.g. you're
> testing a GNU/Linux toolchain with glibc not installed in the target
> system's root directory, you need a board file that sets ldflags to use
> appropriate -Wl,-dynamic-linker -Wl,-rpath option

I understand and agree with that. I hadn't considered the global picture. As a 
matter of fact I have the following in my board file:
set_board_info compiler "[find_gcc]"

> 
> So a generically useful board file describes the "compiler for the board".
> The compiler for the board may or may not be the same as the compiler
> being tested, if the testsuite you're running is a testsuite for a
> compiler at all.  (And if you're running the GCC compat tests you have
> *three* compilers involved - the compiler for the board, the GCC being
> tested and the other compiler you're testing ABI compatibility with.)

Still agree so far.

> 
> The argument for compiling testglue with the "compiler for the board" is
> simply that testglue is not part of the GCC testsuite at all; it's part of
> DejaGnu and needs to be built even in cases where you're running a
> non-compiler testsuite.

This is where I'm not convinced yet. testglue is built explicitly by the GCC 
testsuite (in gcc/testsuite/wrapper.exp) so I think it is reasonable to have a 
behavior specific to the fact that a toolchain is being tested. Using the tested 
toolchain to build testglue seems to be like one more test for the toolchain.

It would also provide some consistence with in tree testing where one cannot 
assume a toolchain is already available. So while I agree dejagnu can assume a 
toolchain is in the PATH, the testsuite shouldn't make this assumption, ie I 
think it should invoke build_wrapper such that the tested toolchain will be used.

Does it make sense?

Best regards,

Thomas
>