From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <gcc-return-42509-listarch-gcc=gcc.gnu.org@gcc.gnu.org>
Received: (qmail 25537 invoked by alias); 9 Dec 2001 21:02:14 -0000
Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm
Precedence: bulk
List-Archive: <http://gcc.gnu.org/ml/gcc/>
List-Post: <mailto:gcc@gcc.gnu.org>
List-Help: <http://gcc.gnu.org/ml/>
Sender: gcc-owner@gcc.gnu.org
Received: (qmail 25516 invoked from network); 9 Dec 2001 21:02:13 -0000
Received: from unknown (HELO nile.gnat.com) (205.232.38.5)
  by sources.redhat.com with SMTP; 9 Dec 2001 21:02:13 -0000
Received: by nile.gnat.com (Postfix, from userid 338)
	id BED89F28F1; Sun,  9 Dec 2001 16:01:28 -0500 (EST)
From: dewar@gnat.com
To: dewar@gnat.com, zack@codesourcery.com
Subject: Re: ACATS legal status cleared by FSF
Cc: gcc@gcc.gnu.org, kenner@vlsi1.ultra.nyu.edu, mrs@windriver.com
Message-Id: <20011209210128.BED89F28F1@nile.gnat.com>
Date: Sun, 09 Dec 2001 14:00:00 -0000
X-SW-Source: 2001-12/txt/msg00445.txt.bz2

<<I think this may be the crux of the difference between the B tests and
the existing "noncompile" tests for gcc and g++.  We - all the people
arguing for inclusion of noncompile tests - are used to a context
where it is easy to automate verification that diagnostics are
correctly issued.  When diagnostics do legitimately change, the test
harness has to be adjusted, but this is straightforward, easily done
by the person who changed the diagnostics.
>>

I don't think the noncompile tests for g++ are anything like as
comprehensive as the B tests in ACATS, which go out of there way
to test every marginal condition in the ARM. The history of these
tests is that they are done sentence by sentence against the RM,
probably there are 50,000 separate tests in all or something like
that, since many tests contain dozens of errors, and there are
thousands of tests. Furthermore, the tests were specifically
designed to check marginal cases (boundary condition testing
was the philosophy of the ACVC tests in the first place). A
consequence is that when a message changes, or disappears, or
moves to a different location, it often takes quite a bit of
expertise in the detailed semantics of Ada at the RM level to
determine whether the change is legitimate. Note that an incorrect
change to the base line, which might be of little consequence in
the g++ case, can be a serious bug in the Ada case, since it
could cause a failed validation in the future, so these baseline
changes have to be done with extreme care.

I certainly think we should upload the B tests, and we have no
problem submitting the current baselines, and people are welcome
to see whether changes they make make a difference, but I think
it is an unnecessary burden on people to require that these tests
be run, and certainly an unneccessary burden to require that the
baselines be updated.

I think the concern here is the following. The question of whether
to require/recommend/suggest that the Ada test suite be run as part
of major/minor gcc bugfix/newfeature modifications is one that needs
discussing, but I (and others familiar with the B tests) feel that
it is far better to encourage people to run the C tests, which are
likely to be far more useful, as well as executable tests that we will
provide to supplement these tests, and have more people doing this,
than having fewer people run the more onerous B tests. And certainly
the L tests should be abandoned as per previous discussion of the
subject.

I will ask Gail Schenker to provide Laurent with the current B test
baselines, and then he can do with them as he sees fit.