Test suite results what they mean and how to evaluate

Test suite results what they mean and how to evaluate

[Mark Henwood]

I have just finished building gcc 4.8.2 so that I can setup a cross
compilation environment.  Running the test suite generated a few errors and
failures but I am not sure how to evaluate them.  Here are the results I
got:

gcc.log:ERROR: Failed to mmap
gcc.log:FAIL: varl is -1, not 6
g++.log:ERROR: Failed to mmap
g++.log:FAIL: varl is -1, not 6
g++.log:FAIL: g++.old-deja/g++.mike/temp.C -std=c++98 execution test
g++.log:FAIL: g++.old-deja/g++.mike/temp.C -std=c++11 execution test
libmudflap.log:FAIL: libmudflap.c++/pass41-frag.cxx ( -O) execution test
libmudflap.log:FAIL: libmudflap.c++/pass41-frag.cxx (-O2) execution test
libmudflap.log:FAIL: libmudflap.c++/pass41-frag.cxx (-O3) execution test

In looking at the results on  http://gcc.gnu.org/buildstat.html it is hard
to make much of the results.  The "best" X86_64 4.8.2 build had just the
libmudflap errors, but similar test results had hundreds of errors.  
As to my results, I can say that every X86_64 4.8.2 build I looked at failed
the libmudflap pass41-frag.cxx test so I am guessing they are not a problem.  

I am not as sure of the others:

Result                           Test
------                           -----
Failed to mmap               
.../gcc/testsuite/c-c++-common/asan/rlimit-mmap-test-1.c
varl is -1                       
.../gcc/testsuite/gcc.dg/guality/guality.exp 
...c++98 execution tests.  g++.old-deja/g++.mike/temp.C -std=c++98 execution
test
...c++11 execution tests.  g++.old-deja/g++.mike/temp.C -std=c++11 execution
test

My questions are 
1. Are these failures a problem?
2. How can I find out what a particular test is for?
3. Should I submit my results to the buildstat people?

Thanks Mark




--
View this message in context: http://gcc.1065356.n5.nabble.com/Test-suite-results-what-they-mean-and-how-to-evaluate-tp1001613.html
Sent from the gcc - Help mailing list archive at Nabble.com.

Re: Test suite results what they mean and how to evaluate

[Ian Lance Taylor]

On Fri, Jan 10, 2014 at 7:49 AM, Mark Henwood <mhenwood@ieee.org> wrote:
>
> I have just finished building gcc 4.8.2 so that I can setup a cross
> compilation environment.  Running the test suite generated a few errors and
> failures but I am not sure how to evaluate them.  Here are the results I
> got:
>
> gcc.log:ERROR: Failed to mmap
> gcc.log:FAIL: varl is -1, not 6
> g++.log:ERROR: Failed to mmap
> g++.log:FAIL: varl is -1, not 6
> g++.log:FAIL: g++.old-deja/g++.mike/temp.C -std=c++98 execution test
> g++.log:FAIL: g++.old-deja/g++.mike/temp.C -std=c++11 execution test
> libmudflap.log:FAIL: libmudflap.c++/pass41-frag.cxx ( -O) execution test
> libmudflap.log:FAIL: libmudflap.c++/pass41-frag.cxx (-O2) execution test
> libmudflap.log:FAIL: libmudflap.c++/pass41-frag.cxx (-O3) execution test



> I am not as sure of the others:
>
> Result                           Test
> ------                           -----
> Failed to mmap
> .../gcc/testsuite/c-c++-common/asan/rlimit-mmap-test-1.c
> varl is -1
> .../gcc/testsuite/gcc.dg/guality/guality.exp
> ...c++98 execution tests.  g++.old-deja/g++.mike/temp.C -std=c++98 execution
> test
> ...c++11 execution tests.  g++.old-deja/g++.mike/temp.C -std=c++11 execution
> test
>
> My questions are
> 1. Are these failures a problem?
> 2. How can I find out what a particular test is for?
> 3. Should I submit my results to the buildstat people?


Ignore all libmudflap failures.  You aren't going to use it, and it's
going away.

Ignore all guality errors, they are very sensitive to the exact
version of gdb you happen to have installed.

The other errors are worth investigating.  I would not expect to see
them in a released version of GCC.  They don't necessarily indicate a
real problem, but you should check to make sure.

The name of the test failure is a file under gcc/testsuite.  Look at
that file and see if it does anything that might be expected to fail
in your environment.  For example, g++.old-deja/g++.mike/temp.C
expects getenv to return something meaningful.  You mentioned you have
a cross-compilation environment; perhaps getenv doesn't do what the
test expects.  Similarly the "failed to mmap" error may be a result of
your environment, in which case you may be able to ignore the test
failure.

Ian

Re: Test suite results what they mean and how to evaluate

[Mark Henwood]

Thanks Ian,

After looking at the mmap test the return was ERROR not FAILED and
apparently that is what was expected.


class T {
  int i;
public:
  T() {
    i = 1;
    printf("T() at %x\n", this);
  }
  T(const T& o) {
    i = o.i;
    printf("T(const T&) at %x <-- %x\n", this, &o);
  }
  T operator +(const T& o) {
    T r;
    r.i = this->i + o.i;
    return r;
  }
  operator int () {
    return i;
  }
  ~T() { printf("~T() at %x\n", this); }
} s, b;

int foo() { return getenv("TEST") == 0; }

int main() {
  printf("foo()=%d\n",foo());
  int i = foo() ? s+b : s;
  printf("i==%d\n",i);
  return i != 2;
}


The temp.C -std=c++98 & -std=c++11 I don't really understand.  I pulled
test.C out, compiled it with the info I pulled out of the logs, set the
LD_LIBRARY_PATH and ran it.  The result is 2 when the TEST environment
variable is not set and 1 when it is (I added the printf statements to
main.)

This only leaves me with the question "What is this testing?" 
I assumed it was a test to see if the compiler recognized some particular
syntax, but this code's output seems to be determined by the TEST
environment variable.  Which the code does not check TEST before hand but
seems to expect it to be set.  This may be valid for the expected
environment.

Mark
   



--
View this message in context: http://gcc.1065356.n5.nabble.com/Test-suite-results-what-they-mean-and-how-to-evaluate-tp1001613p1001742.html
Sent from the gcc - Help mailing list archive at Nabble.com.