Running gcc testsuite outside of gcc's sourcetree.

Running gcc testsuite outside of gcc's sourcetree.

[Nicolas Noble]

Hello,

  Long story short, I'm looking for a way to test a distribution's
compiler by running the latest gcc testsuite on it, but so far, I've
only seem to run it on the same gcc sourcetree it's on. I actually
wonder if it's possible and/or relevant to do this on the
distribution's compiler.

  My problem resides in RedHat's gcc (which version seems to be gcc
(GCC) 4.1.2 20071124 (Red Hat 4.1.2-42)). I recently discovered that
this compiler hosts a bunch of known gcc bugs that have been reported
and fixed in the gcc mainstream, but it seems the bugfixes never got
ported back in RedHat's.

  Now only one of these bugs bit me, and bit me hard, and that's how I
discovered the whole thing. I manually ran a bunch of the testcases in
the gcc testsuite, and I already found 15 bugs active there. But
running of all them by hand is difficult and painful, but I don't see
how to do that automatically on the system's compiler from the
testsuite's documentation. My ultimate goal is to evaluate to which
extend I can trust this compiler, and use these results to convince my
management that we're most probably going to face a lot more trouble
if we continue using a compiler that's containing known bugs.

  Is it possible to run the testsuite on the system's compiler ? I've
seen that some of the testcases might not be relevant as they're
trying to check some of gcc's new features that might not be in that
old version. Or would there be another, simpler testsuite I could run
natively ?

  Thanks!

  -- Nicolas Noble

Re: Running gcc testsuite outside of gcc's sourcetree.

[Dave Korn]

Nicolas Noble wrote:

>   Is it possible to run the testsuite on the system's compiler ?

  See contrib/test_installed

    cheers,
      DaveK

Re: Running gcc testsuite outside of gcc's sourcetree.

[Jeff Law]

On 09/19/09 18:14, Nicolas Noble wrote:
> Hello,
>
>    Long story short, I'm looking for a way to test a distribution's
> compiler by running the latest gcc testsuite on it, but so far, I've
> only seem to run it on the same gcc sourcetree it's on. I actually
> wonder if it's possible and/or relevant to do this on the
> distribution's compiler.
>    
It's certainly possible.  Under the hood the Makefile just invokes 
runtest.  I don't think people do this that often anymore, so you might 
have to set some environment variables and/or set some paths, but when 
it's all said and done you can probably do something like

cd <gcc sources>/gcc/testsuite
runtest --tool=gcc
runtest --tool=g++

Which should run the gcc & g++ testsuites with your system compiler.

>    My problem resides in RedHat's gcc (which version seems to be gcc
> (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42)). I recently discovered that
> this compiler hosts a bunch of known gcc bugs that have been reported
> and fixed in the gcc mainstream, but it seems the bugfixes never got
> ported back in RedHat's.
>    
The fact of the matter is that every compiler has bugs.  Backporting 
every bugfix to old releases isn't practical, particularly if the fix 
has the potential to introduce new regressions.  It's a balancing act 
our engineers deal with daily.  Customers (of course) have input into 
how we balance those decisions, and if you are a customer I would 
strongly encourage you to file a bug report with Red Hat.


Jeff