Re: ACATS legal status cleared by FSF

[Geert Bosch <bosch@gnat.com>]

On Wednesday, December 5, 2001, at 06:05 , <guerby@acm.org> wrote:
> Class A tests check for acceptance (compilation) of language
> constructs that are expected to compile without error.
>
> Class B tests  check that illegal  constructs are recognized  and
> treated as fatal  errors. They are  not expected to  successfully
> compile, bind, or execute.
>
> Class C tests  check that executable  constructs are  implemented
> correctly and produce expected results. These tests are  expected
> to  compile,  bind,   execute  and  report   "PASSED"  or   "NOT-
> APPLICABLE".  Each   class  C   test  reports   "PASSED",   "NOT-
> APPLICABLE", or "FAILED" based on  the results of the  conditions
> tested.
>
> Class D tests check that implementations perform exact arithmetic
> on large literal  numbers. These tests  are expected to  compile,
> bind, execute and report "PASSED". Each test reports "PASSED"  or
> "FAILED" based on the conditions tested. Some implementations may
> report errors at compile  time for some of  them, if the  literal
> numbers exceed compiler limits.
>
> Class E tests check for constructs that may require inspection to
> verify. They have special grading criteria that are stated within
> the test source.
>
> Class L tests check that all  library unit dependencies within  a
> program are  satisfied  before  the  program  can  be  bound  and
> executed, that  circularity  among  units is  detected,  or  that
> pragmas  that  apply  to   an  entire  partition  are   correctly
> processed.
>>>
>
> All ACATS tests are identifed by a 7 characters key which is more or
> less composed from Class + Reference Manual Chapter + RM Section + Key.
>
> All the files for a test begin by this 7 character key, most tests
> have only one file, but some have more than one.  An ACATS source file
> can contain multiple compilation unit, to run them through GNAT we
> first need to "gnatchop" them so that each unit ends up with one file
> with the name GNAT expects by default, then we have to guess what is
> the file containing the main routine, "gnatmake" it and run it.
> This increases the count of source files from 2500 to 4100.
>
> To avoid preprocessing and script-like machinery overhead, I propose
> that we commit the ACATS sources directly in the form expected by
> GNAT, so we end up just having a list of main unit names, and so a
> simple minded loop of "gnatmake x; run x" will be the only thing
> needed to run.
>
> [1] Any objection?
I think your idea of having a set of easily/quickly runnable set of
ACATS tests is a good one.
>
> [2] Do we want to keep some subdirectories, if so what granularity, or
> all in one dir is okay (something like 4000 files)?
 From experience I find it really valuable to have separate 
subdirectories
per class/chapter. For example, if you work on some changes to handling
of floating point, you would initially just want to run chapter 
CXG, which
means the executable RM Annex G tests.
> [3] Some file names will be up to 56 characters, okay? (87% have less
> or equal than 14.)
It might be preferable to "krunch" the names to a more managable length
using the -gnatk option, since there are still many filesystems 
that have
issues with long names and it's also easier to deal with shorter names
as human beings (those who have seen the few examples of few long 
names in
ACATS tests, will agree they do not help: only the first few are 
useful).
> [4] What should be the top directory in the gcc tree for the ACATS
> test suite?  gcc/gcc/testsuite/ada/acats or a top level directory or
> something else? I assume we want ACATS somewhat separated from other
> Ada tests generated by the GCC project, this will facilitate
> maintenance.
We should definately keep ACATS completely separate.
> [5] IMHO it is best if the form in which we commit the sources of the
> ACATS tests is as separated as possible from the testsuite harness
> technology at first, just sources, README and a list of test names.
> Okay?
Seems OK.
> There are convoluted tests trying to see if the compiler conforms to
> the Ada compilation model initially suggested by the RM, but as RMS
> invented a new way of reading this and that GNAT does follow it (and
> modern proprietary compiler do the same), these tests are of no value,
> I propose we just drop them. I'll of course document all such tests
> dropped and why
Indeed, I would not bother by these. I do not see any benefit of these
for testing, mostly they just add a lot of script work and clutter for
no gain.
> The B tests require a lot of maintainance (hundreds of pages of
> changes each time you improve a message, split of files with too many
> errors, etc...), and have no value for the backend, may be someone
> will volunteer the packaging, but not me. I assume ACT dedicates
> someone to this task anyway :).
Yes, agreed. These tests just represent a lot of maintenance work for
very little benefit, since these are only tests of programs with errors
and indeed ACT already does this work.
> [7] I can help if someone wants to take over this, but ACT has probably
> to provide an initial baseline of splits and scripts, so please check
> with them first.
Even though these scripts and splits are not secret or anything, they
change rapidly and something given today will be pretty useless a couple
of weeks from now. This would just be too much of a hassle to deal with.
> I have a patch applying to all tests using the delay statement in
> order to be able to rescale them
That's OK. Please use same casing as in rest of the test though.
> [8] I propose to commit directly with the patch applied, I'll of course
> commit and maintain a list of tests that were modified and why, any
> objection?
Fine. I was wondering though, whether it would make sense to bundle 
tests
into bigger batches. There really is no reason why we couldn't have 
tests
that work on an entire chapter at a time by just making a master 
procedure
that calls all other tests, and I'd think that might speed up the 
process.
Did you ever try that, and if so, what were your findings?
> [10] The first harness prototype will be ultra simple and should allow
> anyone to play with it so we can make progress on portability and
> features later on by patches on stuff in CVS the regular GCC way
> instead of by generating a 20 MB tarball each time we change
> something.
You did not really address the reason why you had decided not to use
the existing testing harness. I can see a number of reasons why seperate
scripts would be better for ACATS, but these should be documented.
> The upstream ACATS is maintained using CVS, web documents and
> interface from <http://www.ada-auth.org/>. There's a very low volume
> public mailing list.
>
> [12] Do we want to scrap the legalese and replace it by something else?
> Do we want to identify changes or extension made within the GCC
> project, and if so how?
I wouldn't think so. This seems to only add confusion about the 
licensing.
Of course if we would make significant changes to the tests, we 
should put
the files under GPL. Also any new files and scripts should be GPL-ed.
> [13] I apply for maintainership of this stuff once commited, and will
> keep it in synch with the change made upstream.
This would be great.

   -Geert