From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 31851 invoked by alias); 6 Dec 2001 17:20:54 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 31827 invoked from network); 6 Dec 2001 17:20:53 -0000 Received: from unknown (HELO nile.gnat.com) (205.232.38.5) by sources.redhat.com with SMTP; 6 Dec 2001 17:20:53 -0000 Received: from darwin (DARWIN.GNAT.COM [205.232.38.44]) by nile.gnat.com (Postfix) with ESMTP id 14680F28EC; Thu, 6 Dec 2001 12:20:07 -0500 (EST) Date: Thu, 06 Dec 2001 09:34:00 -0000 Subject: Re: ACATS legal status cleared by FSF Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v475) Cc: gcc@gcc.gnu.org To: guerby@acm.org From: Geert Bosch In-Reply-To: <200112052305.fB5N5D700531@ulmo.localdomain> Message-Id: <9F39173C-EA6D-11D5-8627-00039344BF4A@gnat.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.475) X-SW-Source: 2001-12/txt/msg00287.txt.bz2 On Wednesday, December 5, 2001, at 06:05 , wrote: > Class A tests check for acceptance (compilation) of language > constructs that are expected to compile without error. > > Class B tests check that illegal constructs are recognized and > treated as fatal errors. They are not expected to successfully > compile, bind, or execute. > > Class C tests check that executable constructs are implemented > correctly and produce expected results. These tests are expected > to compile, bind, execute and report "PASSED" or "NOT- > APPLICABLE". Each class C test reports "PASSED", "NOT- > APPLICABLE", or "FAILED" based on the results of the conditions > tested. > > Class D tests check that implementations perform exact arithmetic > on large literal numbers. These tests are expected to compile, > bind, execute and report "PASSED". Each test reports "PASSED" or > "FAILED" based on the conditions tested. Some implementations may > report errors at compile time for some of them, if the literal > numbers exceed compiler limits. > > Class E tests check for constructs that may require inspection to > verify. They have special grading criteria that are stated within > the test source. > > Class L tests check that all library unit dependencies within a > program are satisfied before the program can be bound and > executed, that circularity among units is detected, or that > pragmas that apply to an entire partition are correctly > processed. >>> > > All ACATS tests are identifed by a 7 characters key which is more or > less composed from Class + Reference Manual Chapter + RM Section + Key. > > All the files for a test begin by this 7 character key, most tests > have only one file, but some have more than one. An ACATS source file > can contain multiple compilation unit, to run them through GNAT we > first need to "gnatchop" them so that each unit ends up with one file > with the name GNAT expects by default, then we have to guess what is > the file containing the main routine, "gnatmake" it and run it. > This increases the count of source files from 2500 to 4100. > > To avoid preprocessing and script-like machinery overhead, I propose > that we commit the ACATS sources directly in the form expected by > GNAT, so we end up just having a list of main unit names, and so a > simple minded loop of "gnatmake x; run x" will be the only thing > needed to run. > > [1] Any objection? I think your idea of having a set of easily/quickly runnable set of ACATS tests is a good one. > > [2] Do we want to keep some subdirectories, if so what granularity, or > all in one dir is okay (something like 4000 files)? From experience I find it really valuable to have separate subdirectories per class/chapter. For example, if you work on some changes to handling of floating point, you would initially just want to run chapter CXG, which means the executable RM Annex G tests. > [3] Some file names will be up to 56 characters, okay? (87% have less > or equal than 14.) It might be preferable to "krunch" the names to a more managable length using the -gnatk option, since there are still many filesystems that have issues with long names and it's also easier to deal with shorter names as human beings (those who have seen the few examples of few long names in ACATS tests, will agree they do not help: only the first few are useful). > [4] What should be the top directory in the gcc tree for the ACATS > test suite? gcc/gcc/testsuite/ada/acats or a top level directory or > something else? I assume we want ACATS somewhat separated from other > Ada tests generated by the GCC project, this will facilitate > maintenance. We should definately keep ACATS completely separate. > [5] IMHO it is best if the form in which we commit the sources of the > ACATS tests is as separated as possible from the testsuite harness > technology at first, just sources, README and a list of test names. > Okay? Seems OK. > There are convoluted tests trying to see if the compiler conforms to > the Ada compilation model initially suggested by the RM, but as RMS > invented a new way of reading this and that GNAT does follow it (and > modern proprietary compiler do the same), these tests are of no value, > I propose we just drop them. I'll of course document all such tests > dropped and why Indeed, I would not bother by these. I do not see any benefit of these for testing, mostly they just add a lot of script work and clutter for no gain. > The B tests require a lot of maintainance (hundreds of pages of > changes each time you improve a message, split of files with too many > errors, etc...), and have no value for the backend, may be someone > will volunteer the packaging, but not me. I assume ACT dedicates > someone to this task anyway :). Yes, agreed. These tests just represent a lot of maintenance work for very little benefit, since these are only tests of programs with errors and indeed ACT already does this work. > [7] I can help if someone wants to take over this, but ACT has probably > to provide an initial baseline of splits and scripts, so please check > with them first. Even though these scripts and splits are not secret or anything, they change rapidly and something given today will be pretty useless a couple of weeks from now. This would just be too much of a hassle to deal with. > I have a patch applying to all tests using the delay statement in > order to be able to rescale them That's OK. Please use same casing as in rest of the test though. > [8] I propose to commit directly with the patch applied, I'll of course > commit and maintain a list of tests that were modified and why, any > objection? Fine. I was wondering though, whether it would make sense to bundle tests into bigger batches. There really is no reason why we couldn't have tests that work on an entire chapter at a time by just making a master procedure that calls all other tests, and I'd think that might speed up the process. Did you ever try that, and if so, what were your findings? > [10] The first harness prototype will be ultra simple and should allow > anyone to play with it so we can make progress on portability and > features later on by patches on stuff in CVS the regular GCC way > instead of by generating a 20 MB tarball each time we change > something. You did not really address the reason why you had decided not to use the existing testing harness. I can see a number of reasons why seperate scripts would be better for ACATS, but these should be documented. > The upstream ACATS is maintained using CVS, web documents and > interface from . There's a very low volume > public mailing list. > > [12] Do we want to scrap the legalese and replace it by something else? > Do we want to identify changes or extension made within the GCC > project, and if so how? I wouldn't think so. This seems to only add confusion about the licensing. Of course if we would make significant changes to the tests, we should put the files under GPL. Also any new files and scripts should be GPL-ed. > [13] I apply for maintainership of this stuff once commited, and will > keep it in synch with the change made upstream. This would be great. -Geert