[PATCH] Add g++.torture testsuite

[PATCH] Add g++.torture testsuite

[Giovanni Bajo]

[-- Attachment #1: Type: text/plain, Size: 4171 bytes --]

Hello,

I think we want a g++.torture testsuite, for the following reasons:

1) g++.dg is too small. It happens very often that non-trivial fixes to the C++
frontend are not caught by regression tests, and then followed up only thanks
to bug reports generated compiling Boost, POOMA and similar testcases. With a
bigger testcase, we can do changes more confidently than before.
2) Miscompilations in big C++ testcases can be very hard to trace, so we better
catch this kind of regressions sooner than later.
3) By collecting several big C++ testcases, we give handy tests to our tree-ssa
guys to improve optimizers. This very testsuite could be also tested for
compile-time and memory-time regressions in the future (I'm sure that QMTest
could be modified to draw and publish compile/execute graphs on the web, etc.).

I have no clue about how to add a new testsuite, so I just guessed. The
attacched file contains the new files that must be decompressed in the tree, is
there anything else required? I'll prepare a sourcebuild.texi patch to document
this by the way once I am done with my other contribute.html patch (which will
also affect sourcebuild.texi).

g++-torture.exp is a straight copy of c-torture.exp, with a few lines changed
(diff attacched for reference). Since objc-torture.exp does the same, I hope
there are no problems with this.

spirit.C is the first testcase for the testsuite, it is a sample of the
Boost.Spirit library, which implements a parser for the C language, and test it
on an internal test file. The testcase is uninclude-d, that is run through
Alexandre Oliva's uninclude script (which I will be submitting for contrib/
soon): this script processes a preprocessed file and converts back preprocessed
standard includes into #include directives. We basically end up with a single
file which only depends on standard headers. We do not want v3 preprocessed in
the source code because:

- v3 code is easily broken from version to version of GCC, as it's only tested
with GCC itself. Plus, it relies on builtins that tend to change.
- atomicity.h and friends have inline assembly which would make the testcase
totally non portable.

spirit.C for instance can be easily compiled with GCC 3.3.3, 3.4.0 or 3.5.0
without touching it. This wouldn't be possible without unincluding it. Extra
care must also be done to handle standard macros such as assert and offsetof.
These guidelines on creating g++.torture testcases will be documented in my
sourcebuild.texi patch.

OK for mainline?

Giovanni Bajo


testsuite/
        * lib/g++-torture.exp: New file.
        * g++.torture, g++.torture/execute: New directories.
        * g++.torture/execute/execute.exp: New file.
        * g++.torture/execute/spirit.C: New testcase.



--- c-torture.exp       2002-05-25 05:40:20.000000000 +0200
+++ g++-torture.exp     2004-06-04 12:32:48.000000000 +0200
@@ -83,8 +83,8 @@
     set options ""
     lappend options "additional_flags=-w $option"

-    set comp_output [gcc_target_compile "$src" "$output" object $options];
-    gcc_check_compile $testcase $option $output $comp_output
+    set comp_output [g++_target_compile "$src" "$output" object $options];
+    g++_check_compile $testcase $option $output $comp_output
     remote_file build delete $output
 }

@@ -168,9 +168,9 @@
        if { $additional_flags != "" } {
            lappend options "additional_flags=$additional_flags";
        }
-       set comp_output [gcc_target_compile "$sources" "${execname}" executable
$options];
+       set comp_output [g++_target_compile "$sources" "${execname}" executable
$options];

-       if ![gcc_check_compile "$testcase compilation" $option $execname
$comp_output] {
+       if ![g++_check_compile "$testcase compilation" $option $execname
$comp_output] {
            unresolved "$testcase execution, $option"
            remote_file build delete $execname
            continue
@@ -214,7 +214,7 @@
            }
        }
        if { $skip == 0 } {
-           set result [gcc_load "$execname" "" ""]
+           set result [g++_load "$execname" "" ""]
            set status [lindex $result 0];
            set output [lindex $result 1];
        }

[-- Attachment #2: g++t.tar.bz2 --]
[-- Type: application/octet-stream, Size: 58289 bytes --]

Re: [PATCH] Add g++.torture testsuite

[Zack Weinberg]

"Giovanni Bajo" <giovannibajo@libero.it> writes:

> 1) g++.dg is too small. It happens very often that non-trivial fixes
> to the C++ frontend are not caught by regression tests, and then
> followed up only thanks to bug reports generated compiling Boost,
> POOMA and similar testcases. With a bigger testcase, we can do
> changes more confidently than before.
>
> 2) Miscompilations in big C++ testcases can be very hard to trace,
> so we better catch this kind of regressions sooner than later.
>
> 3) By collecting several big C++ testcases, we give handy tests to
> our tree-ssa guys to improve optimizers. This very testsuite could
> be also tested for compile-time and memory-time regressions in the
> future (I'm sure that QMTest could be modified to draw and publish
> compile/execute graphs on the web, etc.).

These are all valid concerns but I don't see why they can't be
addressed by adding more test cases to g++.dg?

zw

Re: [PATCH] Add g++.torture testsuite

[Giovanni Bajo]

Zack Weinberg wrote:

> These are all valid concerns but I don't see why they can't be
> addressed by adding more test cases to g++.dg?

Because compiling and executing those big testcases at different optimization
levels gives unvaluable testing to the frontend (and the optimizers... we know
how C++ files can stress the middle-end). Or am I missing something?

Giovanni Bajo

Re: [PATCH] Add g++.torture testsuite

[Zack Weinberg]

"Giovanni Bajo" <giovannibajo@libero.it> writes:

> Zack Weinberg wrote:
>
>> These are all valid concerns but I don't see why they can't be
>> addressed by adding more test cases to g++.dg?
>
> Because compiling and executing those big testcases at different optimization
> levels gives unvaluable testing to the frontend (and the optimizers... we know
> how C++ files can stress the middle-end). Or am I missing something?

g++.dg iterates over optimization levels already, iirc?

zw

Re: [PATCH] Add g++.torture testsuite

[Mark Mitchell]

Zack Weinberg wrote:

> "Giovanni Bajo" <giovannibajo@libero.it> writes:
> 
> 
>>Zack Weinberg wrote:
>>
>>
>>>These are all valid concerns but I don't see why they can't be
>>>addressed by adding more test cases to g++.dg?
>>
>>Because compiling and executing those big testcases at different optimization
>>levels gives unvaluable testing to the frontend (and the optimizers... we know
>>how C++ files can stress the middle-end). Or am I missing something?
> 
> 
> g++.dg iterates over optimization levels already, iirc?

No, it doesn't.  And, it probably shouldn't -- most G++ test cases are 
for successful compilation of particular C++ constructs, and rarely 
behave differently at varying optimization levels.

There's no reason that g++.dg/opt, say, couldn't iterate over 
optimization levels, and that would make sense.  There's no reason the 
test cases coudln't still be in dg format, though.

Furthermore, we should keep the testsuite as a regression testsuite, 
adding tests for bugs we've fixed.  It's a mistake to try to build a 
comprehensive feature/optimization testsuite into our testsuite; there 
are other parties out there who do a much better job at that then we 
ever will, and adding tests has a non-zero impact on the amount of time 
it takes to fix a bug.

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com