From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 22780 invoked by alias); 6 Dec 2001 22:32:05 -0000 Mailing-List: contact gcc-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Archive: List-Post: List-Help: Sender: gcc-owner@gcc.gnu.org Received: (qmail 22698 invoked from network); 6 Dec 2001 22:31:54 -0000 Received: from unknown (HELO alisier.wanadoo.fr) (193.252.19.55) by sources.redhat.com with SMTP; 6 Dec 2001 22:31:54 -0000 Received: from amyris.wanadoo.fr (193.252.19.150) by alisier.wanadoo.fr; 6 Dec 2001 23:31:43 +0100 Received: from ulmo.localdomain (193.251.50.176) by amyris.wanadoo.fr; 6 Dec 2001 23:31:28 +0100 Received: (from guerby@localhost) by ulmo.localdomain (8.11.6/8.11.6) id fB6MMkv02776; Thu, 6 Dec 2001 23:22:46 +0100 Date: Thu, 06 Dec 2001 14:32:00 -0000 Message-Id: <200112062222.fB6MMkv02776@ulmo.localdomain> X-Authentication-Warning: ulmo.localdomain: guerby set sender to guerby@acm.org using -f From: To: gcc@gcc.gnu.org Subject: ACATS Reply-to: guerby@acm.org X-SW-Source: 2001-12/txt/msg00316.txt.bz2 [I have ISP problems at the moment, I hope this will get through.] [0] General: Jerry: Basically the question is if you want to see the test suite as an scaled down ACATS (in which case all changes etc need to be documented), or see ACATS as a starting point, and develop the suite further, for example adding regression testing for discovered bugs, etc. My first impulse is to stick to ACATS, to follow language development, and add extra tests separately. self: ACATS in GCC will be the part of ACATS where a volunteer has done the packaging work. I proposed to do a piece of this packaging, if people want to work on packaging more I'll try to help but won't do it myself, I certainly agree that packaging all of ACATS is a great goal, as I said I'll carefully document what I left out and why, anyone is free to finish the work. IMHO non ACATS or ACATS-derived tests should be packaged in another subdirectory so that it is easier to track upstream ACATS. The December ACATS update email has been sent, I'm forwarding it to the list so people can make an idea. Following upstream ACATS should be very easy, all changes are very well documented and there isn't a large volume of changes. Joseph S. Myers: We ought to have a link to this from the Ada section of readings.html. self: I'll work hard at documenting and linking to a maximum of Ada ressources, Ada is pretty unique in that all (or nearly all) language requirement, design, norms, maintainance and testsuite sources and documents are freely available. [1] pre-gnatchop Jerry: I assume we want to run an ACATS subset as a basic sanity check, right ? But even in that case, it would be advisable to follow the changes made to the test suite. Your proposal means this has to be done manually by someone. Who is going to monitor this, and propagate all changes in the tests ? self: I proposed to do the upstream tracking work for the executable section. As for the subset we want to run, this is open to discussions, I have no opinion on the subject. Personally, I'll run every single GCC test available for all of my changes up to 8 hours of wall clock time (make a daily patch, check during night, commit in the morning). [2] directory structure Mike Stump: Split them up some. <200 per directory would be good (or maybe <359 per directory). Jerry: I like the current subdirectory structure, it makes it easier if you want to check just one test, or a set or related tests. Geert: From experience I find it really valuable to have separate subdirectories per class/chapter. For example, if you work on some changes to handling of floating point, you would initially just want to run chapter CXG, which means the executable RM Annex G tests. self: okay, I'll keep the existing ACATS directory structure. Running easily subsets of scripts is on the spec of any testing technology and shouldn't be related to directory structure. I intend to have the test list file provide test key annotated with a few attributes, like wether the test use tasking, is ok for the no-run-time mode, has some real code, is part of minimal testing requirement, etc... so that it is easy select a test subset. [3] filename length Geert: It might be preferable to "krunch" the names to a more managable length using the -gnatk option, since there are still many filesystems that have issues with long names and it's also easier to deal with shorter names as human beings (those who have seen the few examples of few long names in ACATS tests, will agree they do not help: only the first few are useful). self: I believe the longest name is ca13001_1-ca13001_2-ca13001_4-family_transportation.adb There is a test checking for the longest package name supported by the implementation, this is part of my "drop because stupid and painful" list. Such tests should be auto-generated and tailored to GCC in some other part of the test suite, the ACATS stuff is of little value. Mike Stump: I'd say ok. self: let's say okay. [4] top dir gcc/gcc/testsuite/ada/acats Joseph S. Myers: That seems plausible - presumably "runtest --tool ada" will be used in "make check" to run both those and other Ada tests? self: to be decided. [5] source separate from harness self: no comment, okay. [6] drop compilation model tests self: Geert and Robert agreed, okay. [7] B tests Mike Stump: They can be made to require less, by having them line resolution only, this is what the C++ testsuite does. This is for the Ada maintainers to determine. If they want the testing, and want the burden, then they should stay the same, if they want to escape it, then a line based approach can be used. Jerry: I think it's important to test the whole GNAT system, not just the backend. The value of the B tests is that it alerts you to unexpected changes. I think that at least for now they should stay. We can add 'skip tests' for changes that are to much work for the added value. Zack Weinberg: I disagree in the strongest possible terms. Put the B tests in the public repository. If you don't, you only make life harder for people outside of ACT who wish to work on the Ada front end. The maintenance work has to be done anyway, and ought to be the responsibility of the person who makes the change that causes the tests to regress. If the B tests are run as part of "make check" in the FSF tree, this will be enforced automatically. self: B tests are 1525 files, 11886 errors to be flagged (errors are marked with a -- ERROR: xxx comment). No one is against inclusion, I just points out that there is painful initial work and maintainance, and that I don't volunteer to do it, given passionate answers, I assume that Zack and Jerry volunteered to do the work :). I will also point out that actually including this might make it less likely that people will propose patches to improve error messages, since any of such patches will probably generate hundreds of changes to be validated in the B tests, this is not really fun, so by this effect including B test requirement might *lower* the future quality of the compiler by preventing some useful patches. Also note that GNAT currently generates the best error messages I've ever seen for any compiler and language, and by a fair margin. [8] tasking delay patch Geert and Richard Kenner: Test timings? self: ACATS timing on a 1GHz P3 at -O0, in number of test and seconds per section: N ; T(s) ; Desc. 4 ; 30 ; support + cz Check that the test reporting stuff works 75 ; 38 ; a 34 ; 15 ; c2 Lexical Elements 351 ; 267 ; c3 Declarations and Types 339 ; 186 ; c4 Names and Expressions 95 ; 59 ; c5 Statements 81 ; 43 ; c6 Subprograms 51 ; 48 ; c7 Packages 140 ; 141 ; c8 Visibility Rules 255 ; 307 ; c9 Tasks and Synchronization 74 ; 49 ; ca Program Structure and Compilation Issues 43 ; 26 ; cb Exceptions 117 ; 78 ; cc Generic Units 173 ; 87 ; cd Representation Issues 268 ; 298 ; ce Predefined Language Environment (Ada 83) 87 ; 133 ; cxa Predefined Language Environment (Ada 95) 30 ; 32 ; cxb Interface to Other Languages 13 ; 30 ; cxc Systems Programming 38 ; 75 ; cxd Real-Time Systems 1 ; 1 ; cxe Distributed Systems 20 ; 24 ; cxf Information Systems 29 ; 63 ; cxg Numerics 4 ; 19 ; cxh Safety and Security 4 ; 2 ; d 11 ; 5 ; e 26 ; 19 ; l Joseph S. Myers: It is arguable that perhaps the CVS vendor branch mechanism should be used - import the unmodified sources on the vendor branch with GCC's modified version on the mainline. This should make it easy to use CVS to see what the differences from unmodified ACATS are. I don't know whether whatever tests it is deliberately decided to drop rather than include in the GCC testsuite should also go on the vendor branch, or whether it should just be documented which are dropped. self: I'm no CVS specialist, keeping things in ACATS original form will require maintaining piles of dubious scripts at best, I don't want to do that myself, I by far prefer the hand crafted approach, and have things directly in GCC usable form in CVS. [9] prototype tarball ftp upload self: I'll put stuff in [10] first harness prototype Geert: You did not really address the reason why you had decided not to use the existing testing harness. I can see a number of reasons why seperate scripts would be better for ACATS, but these should be documented. Geoff Keating: You do plan to use dejagnu to run the tests, correct? self: My initial tarball will only contain readily compilable sources, a list of test main names and attributes as described above, documentation on the test to run and omitted, and a minimalistic Bourne shell script to run the thing. From there, I assume DejaGNU experts will volunteer or guide volunteers on how to best integrate this in the existing Makefile + dg scripts. I must admit I'm not really impressed by the existing testing framework, and I believe and I can do much better with not that much effort, I'll propose something later on, but I don't want this to delay inclusion of ACATS in usable form in the GCC project, and I certainly don't want to stop people that would volunteer on integrating ACATS and existing DejaGNU. People will also have to decide what is officially part of the mandatory testing process. [12] DFAR legalese and GCC changes Geert: I wouldn't think so. This seems to only add confusion about the licensing. Of course if we would make significant changes to the tests, we should put the files under GPL. Also any new files and scripts should be GPL-ed. self: when I'll gnatchop the original sources, only one file will retain the legalese, I assume I don't need to copy it to every file after gnatchop. Mike Stump: No. Hmm, I don't intend to change anything without properly identifying what I've done (like any other change made to any part of GCC), may be I misunderstand what your answer means? [13] maintainership self: I'll add myself in MAINTAINERS for gcc/gcc/testsuite/ada/acats once the discussion is complete and the commit is done. -- Laurent Guerby