* Ada/GCC testing (was: Why not gnat Ada in gcc?)
2000-11-01 21:12 Why not gnat Ada in gcc? Robert Dewar
@ 2000-11-02 13:34 ` Laurent Guerby
0 siblings, 0 replies; 2+ messages in thread
From: Laurent Guerby @ 2000-11-02 13:34 UTC (permalink / raw)
To: dewar; +Cc: dewar, law, gcc, kenner, mrs, rms, rth, guerby
Robert Dewar wrote:
> [...] As I said earlier, the missing link is to make an easy way of
> runnig these tests, and that is one of the jobs on our list!
I'm willing to contribute work here.
My plans (and rationale) for ACATS/GCC (first round, more ambitious
plans can come later) are as follows:
- It is difficult to package the ACATS run if one wants to stay close
to formal validation (what ACT does). However I think being near
formal Ada validation isn't really that useful for GCC testing (and
I'll skip the details about what formal Ada validation is about, and
what steps were made by the validation body to clarify the status of
validated compilers with publically available sources, may be Robert
Dewar will lecture us on the topic ;-). In particular, I plan to drop
completely the "ACATS compliant" build process and pre-gnatchop
everything once and for all and simply build with gnatmake. In the
same line, only tests with overflow checking needed will be compiled
with -gnato, same kind of thing for the dynamic elaboration madness
test. Tests that do not need checks will be compiled with -gnatp
(optionally). This will bring the testing closer to real world use
IMHO.
- From the point of view of GCC, tests can be classified in two
categories: test of compilers rejecting code with standard violation
(errors) - the "B" tests, test of compiler generated code run as
expected by the standard - the "C" tests. Amongst the "C" tests we can
subvide in two kind of tests: those that need extensive runtime/OS
support (tasking, distributed systems), let's call them "C-runtime"
and those that would run without too much of it - "C-bare" (I include
here exceptions, but I may subclassify if it turns out that these
tests are not easily runnable). I think the most immediately useful
tests to run are the "C-bare" bits, the one that will exercise the
back-end and that should run without a full runtime GNAT port (ie on
platforms like Alpha/GNU/Linux or PowerPC/GNU/Linux where GNAT is not
readily available) "B" tests maintenance is a huge burden (splits due
to too much errors and message changes) and these tests are mostly
front-end. So in a first step, I would not worry about the "B" tests
at all, separate cleanly the "C-runtime" and "C-bare", and concentrate
on getting "C-bare" as easy to run and analyze as
possible. "C-runtime" will be worked upon later on.
- I think ACT does not run ACATS with optimization (always use the
validation flags and -O0, someone from ACT?), it should be easy to turn
that on (may be some test cleaning will be needed, we'll see). Also
test with -fPIC since shared libraries are fashionable these days.
- I will write a small Ada driver for the testing process (build, run,
check, report various gathered data in text + spreadsheet + HTML) so
that no other tool is needed to run the thing (other than tools to
build the compiler itself anyway).
- There seems to be some RTEMS/GNAT interest on the RTEMS mailing
list, I hope people will contribute some bits to make the thing
runnable in cross compilation environments.
- I will run the tests against GNAT 3.13p and public releases. Once
GNAT is integrated in GCC pre-3.0, and if it reaches 100% pass (it
should since ACT maintains this status AFAIK), I hope the GCC 3.0
release criteria will say that no regressions are allowed ;-).
This is for ACATS, and may be for the tests ACT can release free of
NDA (I think these are the 0000-xxx tests, someone from ACT?). I don't
think ACATS is that useful for back-end testing but it's freely
available (BTW can someone from ACT confirm this), has a pretty wide Ada
standard coverage, and is definetly not insignificant.
There are some Ada performance test suites available, the biggest is
ACES (> 600KSLOC of comment free code, an amazing bazar of algorithms
from all CS fields), there are also some other test smaller suites
Robert Dewar mentionned.
I'm not sure about the copyright status of ACES, I wasn't able to find
the copyright notice (the only one I found is about the "zip" software
used to decompress the downloadable files). I would appreciate if
someone knowledgeable could check what the real status is (someone from
ACT?).
Anyway, ACES comes with its own tool to preprocess, run and analyze
the tests, I didn't find it very useful (in particular it makes no use
of command line switches since this was not available in the Ada 83
standard, and the ASCII screen based selection process is pretty
akward by modern standards). Here also I plan to pre-gnatchop things
and use a simple Ada driver (the same as for ACATS), with
command line switches thanks to Ada 95 ;-).
I already did some things with ACES, my plan is to package the
"relevant" parts of ACATS and ACES for the current GNAT public
version, and it should be done by november 19th, available as
standalone tarballs somewhere on the net.
I'd like to make sure that I do not duplicate ACT work here before
working furiously on it this week-end (weather sucks badly now in
Paris ;-).
If ACT hasn't started on it, could ACT send me the ACATS configuration
bits ACT uses for GNU/Linux on IA32? If official or pending
modifications are not publically available, could ACT send them to me
too? (I can deal with the whole tarball if it's simpler for ACT ;-).
Of course comments and offer for help are welcomed.
PS: I think one neat thing would be to test with gcov what bits of the
front-end and back-ends are exercised during a bootstrap + ACATS +
ACES, but the analysis of the gcov output for such a big piece of
software (GCC+GNAT) will be for sure a big task on its own ;-).
--
Laurent Guerby <guerby@acm.org>
^ permalink raw reply [flat|nested] 2+ messages in thread