Re: ACATS legal status cleared by FSF

Re: ACATS legal status cleared by FSF

[Richard Kenner]

    This is all fair.  How about a compromise position where you check the
    B tests into the CVS tree but don't bother adding dejagnu harnesses
    for them.  They are then instantly available if they are needed.

The issue is the *baselines*, not either the B tests themselves or the
test harness.  They must be checked in *and* actively maintained if
they are to be "instantly available".

One possibility is to run the B tests in a non-conventional way, where all
the test harness does is to check for the presence of at least one
error line for each line marked ERROR.  I don't know how hard such a
harness is to write, and that's not the way B tests are usually done, but
might work.

That would just leave the issue of splits, not baselines, and those change
less often.

ACATS B tests

[guerby]

<http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ACATS/>
will give to anyone access to the ACATS sources from
web, tarballs are at 
<http://www.adaic.org/compilers/acats/2.4/tars/>

I don't remember how the B tests are run at ACT (gnatchop, flags,
multi-file tests if any, big split list, etc...) but I remember it's
much more complicated than executable tests and requires maintainance
of a baseline. In particular, just testing that errors are reported on
lines flagged ERROR will not work for at least half of the tests,
if you bother to look at the first B test:

<http://www.ada-auth.org/cgi-bin/cvsweb.cgi/ACATS/B2/B22001A.TST?rev=1.2>

Commiting the B tests as is in GCC CVS is of zero value given they are
readily available to anyone willing to spend more than one useful
second of actual work on it.

While waiting for someone volunteer to do the B test work instead of
just talking ("talk is cheap" as once said a famous participant of
this list :), I assume it is reasonable to proceed with the executable
part as discussed and so far agreed upon.

I'll upload a prototype tarball this weekend, I have a few hours of
work to do to properly match and package the current upstream ACATS
and write all the documentation I promised :).

-- 
Laurent Guerby <guerby@acm.org>

Re: ACATS B tests

[Richard Kenner]

    In particular, just testing that errors are reported on lines flagged
    ERROR will not work for at least half of the tests, if you bother to
    look at the first B test:

OK, I give up.  Why is that?  Is it because the error was supposed to be
on the *previous* line?

BTW, what is your plan with regard to macro substitutions?  I assume you'll
remove the CE tests that need absolute filenames, right?

Re: ACATS B tests

[guerby]

> OK, I give up.  Why is that?  Is it because the error was supposed to be
> on the *previous* line?

My rough estimate is that yes, about half of errors will be generated
not on the line flagged as ERROR, but I haven't looked at the whole B
tests, just enough to realize that it is a huge task.

> BTW, what is your plan with regard to macro substitutions?  I assume you'll
> remove the CE tests that need absolute filenames, right?

I believe the 3 CE tests doing smart things with filenames have been
removed, I have to recheck otherwise I won't bother with them.

I plan to commit with all macro substituted, most macro deal with the
testsuite being tailored to one special compiler, and we have only one
compiler out there :). GNAT handles platform specific stuff using
Standard'Attribute in a uniform source way, no need for target macros
as far as ACATS for GCC is concerned.

-- 
Laurent Guerby <guerby@acm.org>

Re: ACATS B tests

[dewar]

<<OK, I give up.  Why is that?  Is it because the error was supposed to be
on the *previous* line?
>>

Nope. Not necessarily, for example there are cases where the ACVC test
expects the error message on the definition instead of the reference
or vice versa, and there are cases where the ACVC expects a cascaded
error where we deliberately suppress it, and there are cases where
GNAT generates cascaded errors where none are expected etc etc.