GCC regression testing

GCC regression testing

[Ben Elliston]

I've been following the discussion about the lack of GCC regression
testing and decided to take on the task.  I chose to not use the
existing regression testing scripts, because I think one of the things
that has been lacking is a centralised test results, where useful
analayses can be performed.  The testing is currently being performed
using Red Hat's internal build system, however I hope that it will be
made more accessible in the near future.

For now, I am publishing statically generated Tinderbox-style build
reports every four hours on:

        http://people.redhat.com/bje/tinderbox.html

Testing is being done for the following native configurations:

        i686-pc-linux-gnu
        powerpc-unknown-linux-gnu
        sparc-sun-solaris2.8

This tool is in its infancy and I am working on making it more useful.
As such, I welcome feedback from other GCC developers on how to make
it more useful.

Ben

Re: GCC regression testing

[Janis Johnson]

On Tue, Oct 22, 2002 at 05:51:48PM +1000, Ben Elliston wrote:
>
> This tool is in its infancy and I am working on making it more useful.
> As such, I welcome feedback from other GCC developers on how to make
> it more useful.

Will you be sending mail about regressions to people who made recent
changes to the compiler, as Geoff Keating's regression tester did?

The list of test failures at your web page only shows C tests; are you
building the other compilers as well?

Several people are submitting test results regularly to gcc-testresults,
and I find the archive of that list very useful.  Robert Schiele has his
set up to report bootstrap failures.  Besides reporting your own testing
you might find a way to summarize the information from that archive,
which includes a wide variety of platforms.

Thanks for tackling this.

Janis Johnson
IBM [GNU/]Linux Technology Center

Re: GCC regression testing

[bemis]

Ben,

this is just a sample of our work...it also is still in its infancy since
no one but us currently uses it.
http://astro.iol.unh.edu/work/demo.php?select=all&date=2002-08-26

we too are going to start submitting our regression testing result for
alpha.
we have created a PHP page for displaying multiple architectures and links
to a ftp site to view the checks, and builds.
This example page compares alpha builds with similar intel builds and vice
versa, and compiles statistics based upon make check.

I'd be interested in how you are generating your results for regressions,
and sending them out.  Is there a format that you are going to follow?

Matt Bemis



Janis Johnson wrote:

> On Tue, Oct 22, 2002 at 05:51:48PM +1000, Ben Elliston wrote:
> >
> > This tool is in its infancy and I am working on making it more useful.
> > As such, I welcome feedback from other GCC developers on how to make
> > it more useful.
>
> Will you be sending mail about regressions to people who made recent
> changes to the compiler, as Geoff Keating's regression tester did?
>
> The list of test failures at your web page only shows C tests; are you
> building the other compilers as well?
>
> Several people are submitting test results regularly to gcc-testresults,
> and I find the archive of that list very useful.  Robert Schiele has his
> set up to report bootstrap failures.  Besides reporting your own testing
> you might find a way to summarize the information from that archive,
> which includes a wide variety of platforms.
>
> Thanks for tackling this.
>
> Janis Johnson
> IBM [GNU/]Linux Technology Center

Re: GCC regression testing

[Ben Elliston]

Hi Janis,

>>>>> "Janis" == Janis Johnson <janis187@us.ibm.com> writes:

  Janis> Will you be sending mail about regressions to people who made
  Janis> recent changes to the compiler, as Geoff Keating's regression
  Janis> tester did?

That is the intention.  I'm not at that part yet, but it will be
relatively easy to do in this framework.

  Janis> The list of test failures at your web page only shows C
  Janis> tests; are you building the other compilers as well?

The tests are run for all front-ends.  Those lists are not a list of
failures -- they're a list of regressions.

  Janis> Thanks for tackling this.

My pleasure? :-)

Cheers, Ben

Re: GCC regression testing

[Gerald Pfeifer]

On 22 Oct 2002, Ben Elliston wrote:
> I've been following the discussion about the lack of GCC regression
> testing and decided to take on the task.

Great!

>         http://people.redhat.com/bje/tinderbox.html

We should really document this, so I went ahead an committed the following
patch:

  Do not refer to http://gcc.gnu.org/testresults/ any longer, which has
  not been updated for more than a year.  Add a new page on testers instead.

This is just a skeleton, Ben, feel free to extend it as you see fit.

And, of course, everyone else operating similar testers, please add
yourself as well (or send me some details and I will do it for you).

Thanks,
Gerald

Index: index.html
===================================================================
RCS file: /cvs/gcc/wwwdocs/htdocs/index.html,v
retrieving revision 1.336
diff -u -3 -p -r1.336 index.html
--- index.html	4 Oct 2002 13:38:46 -0000	1.336
+++ index.html	26 Oct 2002 10:09:21 -0000
@@ -128,10 +128,11 @@ snapshots</a>.</p>

 <p>We strive to provide regular, high quality
 <a href="releases.html">releases</a>, which we want to work well on a variety
-of native (including GNU/Linux) and cross targets. To that end, we use an
-extensive <a href="http://gcc.gnu.org/testresults/">test suite</a> as well
-as <a href="benchmarks/">various benchmark suites
-and automated testers</a> to maintain and improve quality.</p>
+of native (including GNU/Linux) and cross targets.
+To that end, we use an extensive <a href="http://gcc.gnu.org/testing/">test
+suite and automated regression testers</a> as well
+as <a href="benchmarks/">various benchmark suites and automated testers</a>
+to maintain and improve quality.</p>


 <!-- NOTE: Of the following four items, we always want to show exactly
Index: testing/index.html
===================================================================
RCS file: testing/index.html
diff -N testing/index.html
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ testing/index.html	26 Oct 2002 10:09:21 -0000
@@ -0,0 +1,15 @@
+<html>
+
+<head>
+<title>GCC Regression Testing</title>
+</head>
+
+<body>
+
+<p>Ben Elliston operates a Tinderbox-style
+<a href="http://people.redhat.com/bje/tinderbox.html">build and regression
+testing system</a>.</p>
+
+</body>
+
+</html>

Re: GCC regression testing

[Benjamin Kosnik]

Dude! You rule.

It's great to see somebody taking the initiative here. It's also great
to get some test results up for these three targets.

Is there anyway to see testsuite results for these three targets? I'd
like to be able to see the g++.log and libstdc++-v3.log, for instance.
Is there a way to do this?

-benjamin