Automatically testing gcc

Automatically testing gcc

[Dimitrios Apostolou]

Hello list,

I've two questions regarding testing:

First, how can I make sure that my change did not affect code generation 
at all? Ideally I would like to run the test-suite on trunk, then on the 
patched version, and diff all object files. Is there an automated way?

Also is it guaranteed that two runs on all testsuite will always produce 
the same code? Or are there maybe some passes that don't guarantee 
determinism?

Second, given that I made a tiny change in the logic, can I make sure that 
there is no case that would generate worse code? Is there automated way 
for that? Does it involve benchmarking or somehow measuring code quality?


Thanks in advance,
Dimitris

Re: Automatically testing gcc

[Paolo Bonzini]

On Sat, Aug 13, 2011 at 03:09, Dimitrios Apostolou <jimis@gmx.net> wrote:
> Hello list,
>
> I've two questions regarding testing:
>
> First, how can I make sure that my change did not affect code generation at
> all? Ideally I would like to run the test-suite on trunk, then on the
> patched version, and diff all object files. Is there an automated way?

There are automated scripts for .s comparison across all the testsuite
but I don't have them.  Usually I just try a bunch of .i files (from
cc1 or elsewhere) and compare the output of those.

> Also is it guaranteed that two runs on all testsuite will always produce the
> same code? Or are there maybe some passes that don't guarantee determinism?

No, that would be a bug.

> Second, given that I made a tiny change in the logic, can I make sure that
> there is no case that would generate worse code? Is there automated way for
> that? Does it involve benchmarking or somehow measuring code quality?

There is SPEC, but it's proprietary and expensive.

Paolo