Re: ACATS legal status cleared by FSF

[Geert Bosch <bosch@gnat.com>]

On Sun, 9 Dec 2001 guerby@acm.org wrote:
  BTW is there any record of the existing noncompile testsuite catching
  problems, or did it just prevent any serious error message work by
  scaring people?

It has not really been useful in catching problems in error messages.
One of the reasons is that most error messages emitted by the compiler
are not triggered at all by the ACATS test. For example, warnings are 
not tested at all and turned off, since the standard doesn't say anything 
about warnings. Also we have for many years collected all reports from 
users who were confused by error messages, and we have tried to improve 
the messages. Exactly these messages and warnings are hardest to get right
and are the ones that do not get tested by ACATS.

In comparison with the ACATS test, regression tests of actual code with 
actual errors that people have made have been far more helpful as tests.
Most advanced error handling and recovery (look for example at the
case of detecting an "is" token replaced by a ";"), is directly
inspired by repeating reports from users. The whole philosophy
behind the error messages in GNAT is not so much to tie them to the 
letter of the RM, but evaluate there usefulness on actual reports
from users.

This is the reason you won't find many direct quotations of
RM rules that have been violated, but rather a less formal message,
which may be a simplification which is clearer for an actual user
(like saying "type T" instead of "the type whose first named subtype is T").
I would encourage anybody (but especially those who have not memorized
the Ada Reference Manual) who finds that GNATemits an unclear error message,
where it clearly could have done better, to file a low priority 
bug report/enhancement request (with complete test case of course)
in the GCC bug database. This kind of reports/tests is what is needed
to improve on error messages. If there is not a to high threshold
in the form of mandatory updating of 50 arcane B test files, somebody
might actually want to improve the message.

  -Geert