'make check' questions

'make check' questions

[Simon Wright]

I see from https://gcc.gnu.org/install/test.html that it's possible to run tests in parallel. I get the impression from gcc/Makefile that the check concerned has to be set up in the Makefile (in my build tree, configured with --target=x86_64-apple-darwin16 --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks and lang_checks_parallelized set empty). So, is it necessary for check-ada or check-acats to cope with being run in parallel (i.e., will they ever see GCC_RUNTEST_PARALLELIZE_DIR set?)

Also in https://gcc.gnu.org/install/test.html, under what circumstances would a test report ERROR (the testsuite detected an error) or WARNING (the testsuite detected a possible problem)? For example, if a particular test that should compile & run has a build error, is that a FAIL or an ERROR? 

Re: 'make check' questions

[Kyrill Tkachov]

On 11/05/17 11:43, Simon Wright wrote:
> I see from https://gcc.gnu.org/install/test.html that it's possible to run tests in parallel. I get the impression from gcc/Makefile that the check concerned has to be set up in the Makefile (in my build tree, configured with --target=x86_64-apple-darwin16 --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks and lang_checks_parallelized set empty). So, is it necessary for check-ada or check-acats to cope with being run in parallel (i.e., will they ever see GCC_RUNTEST_PARALLELIZE_DIR set?)

I don't usually build Ada, but testing with "make -j<n> check" works for me where <n> is the parallelism I want

> Also in https://gcc.gnu.org/install/test.html, under what circumstances would a test report ERROR (the testsuite detected an error) or WARNING (the testsuite detected a possible problem)? For example, if a particular test that should compile & run has a build error, is that a FAIL or an ERROR?

ERROR results are usually problems with the testsuite infrastructure, like misformed DejaGNU directives. They don't usually appear in a clean test run.
If a test fails to build due to a compiler problem i.e. an ICE or other bug it will be a FAIL. If the test harness has a problem with the testsuite directives
syntax I think it will be reported as ERROR.

I usually get WARNINGs if a runtime tests times out. It can happen when testing against simulators or if the test was miscompiled into an infinite loop.

Kyrill

Re: 'make check' questions

[Marek Polacek]

On Thu, May 11, 2017 at 11:50:21AM +0100, Kyrill Tkachov wrote:
> 
> On 11/05/17 11:43, Simon Wright wrote:
> > I see from https://gcc.gnu.org/install/test.html that it's possible to run tests in parallel. I get the impression from gcc/Makefile that the check concerned has to be set up in the Makefile (in my build tree, configured with --target=x86_64-apple-darwin16 --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks and lang_checks_parallelized set empty). So, is it necessary for check-ada or check-acats to cope with being run in parallel (i.e., will they ever see GCC_RUNTEST_PARALLELIZE_DIR set?)
> 
> I don't usually build Ada, but testing with "make -j<n> check" works for me where <n> is the parallelism I want

I have
export MAKEFLAGS='-j4'
in my environment.

	Marek

Re: 'make check' questions

[Simon Wright]

On 11 May 2017, at 11:50, Kyrill Tkachov <kyrylo.tkachov@foss.arm.com> wrote:

> On 11/05/17 11:43, Simon Wright wrote:
>> I see from https://gcc.gnu.org/install/test.html that it's possible to run tests in parallel. I get the impression from gcc/Makefile that the check concerned has to be set up in the Makefile (in my build tree, configured with --target=x86_64-apple-darwin16 --enable-languages=c,c++,ada,fortran,objc,obj-c++ , I see both lang_checks and lang_checks_parallelized set empty). So, is it necessary for check-ada or check-acats to cope with being run in parallel (i.e., will they ever see GCC_RUNTEST_PARALLELIZE_DIR set?)
> 
> I don't usually build Ada, but testing with "make -j<n> check" works for me where <n> is the parallelism I want

So it does. Will have to review the results to see how it's done in the Ada test scripts - deeply unclear to me at the moment.

>> Also in https://gcc.gnu.org/install/test.html, under what circumstances would a test report ERROR (the testsuite detected an error) or WARNING (the testsuite detected a possible problem)? For example, if a particular test that should compile & run has a build error, is that a FAIL or an ERROR?
> 
> ERROR results are usually problems with the testsuite infrastructure, like misformed DejaGNU directives. They don't usually appear in a clean test run.
> If a test fails to build due to a compiler problem i.e. an ICE or other bug it will be a FAIL. If the test harness has a problem with the testsuite directives
> syntax I think it will be reported as ERROR.
> 
> I usually get WARNINGs if a runtime tests times out. It can happen when testing against simulators or if the test was miscompiled into an infinite loop.

OK, I see, thanks.