Testsuite reorganization, documentation, cleanup?

Testsuite reorganization, documentation, cleanup?

[Nathanael Nerode]

Looking at the testsuite, it's rather confusing.  Some directories are
documented in READMEs; many aren't.  There are multiple organizational
systems.  There are multiple drivers, many of which seem very similar;
one suspects redundancy.

I'd like to clean up the organization a little, or at least document it
better.  But that leads to two questions:

1. What do people think the ideal organization should be?

Here's a possible answer:

  One directory for each language's corectness tests, and one for
  anything genuinely non-language-specific (counting preprocessor as a
  language).  

  Beneath that, a subdivision by correctness, code quality, or compiler
  behavior quality.

  Beneath that, subdivision by category where appropriate (abi,
  templates, etc.)

  Beneath that, one directory for each variety (compile, execute, fail to
  compile, produce warning 'blah')

  Name tests with the date you added them plus a digit. (Eek.)

This is obviously not the best arrangement; someone with more experience
can certainly come up with better arrangements.  The point is to have one
organization which encapsulates pretty much everything.

2. What do the various existing undocumented directories represent?

For instance, why are there 'gcc.c-torture', 'gcc.dg',
and 'gcc.misc-tests'?  What are the tests in g++.dg in the subdirectories
'bprob','debug','gcov','opt',and 'tls', and why aren't they mentioned in
g++.dg/README ?  

Why are there 410 tests in gcc.dg, not sorted in any clear fashion?

I'm guessing a lot of this is historical cruft, and *can* theoretically
be simplified.

Help would, well, help. :-)

--Nathanael

Re: Testsuite reorganization, documentation, cleanup?

[Joseph S. Myers]

On Sat, 5 Oct 2002, Nathanael Nerode wrote:

> Looking at the testsuite, it's rather confusing.  Some directories are
> documented in READMEs; many aren't.  There are multiple organizational
> systems.  There are multiple drivers, many of which seem very similar;
> one suspects redundancy.

The documentation is properly in gcc/doc/sourcebuild.texi.  But some
directories are undocumented, and other documentation in READMEs hasn't
yet been moved into the manual.

-- 
Joseph S. Myers
jsm28@cam.ac.uk

Re: Testsuite reorganization, documentation, cleanup?

[Mike Stump]

I think I'd rather leave the old tests where they are.  Sometimes 
people compare them from old releases to new releases, and having name 
stability is sometimes useful.  Id rather someone work on a document 
that describes the `right' way to name testcases going forward, and for 
people to follow that, unless they have specific reasons not to.  I 
have a slight preference for this, I don't feel too strongly about it.  
If someone really wanted to reorganize really poorly done things, like 
the p990912.c type testcases, or stuff in directories called misc, I 
would not object.

On Friday, October 4, 2002, at 09:44 PM, Nathanael Nerode wrote:
> Looking at the testsuite, it's rather confusing.  Some directories are
> documented in READMEs; many aren't.  There are multiple organizational
> systems.  There are multiple drivers, many of which seem very similar;
> one suspects redundancy.

Re: Testsuite reorganization, documentation, cleanup?

[Hans-Peter Nilsson]

On Mon, 7 Oct 2002, Mike Stump wrote:
> I think I'd rather leave the old tests where they are.  Sometimes
> people compare them from old releases to new releases, and having name
> stability is sometimes useful.  Id rather someone work on a document
> that describes the `right' way to name testcases going forward, and for
> people to follow that, unless they have specific reasons not to.

FWIW, I agree, strongly.  I don't want a test-suite flag-day.
I think the cure would be worse than any perceived disease.

> If someone really wanted to reorganize really poorly done things, like
> the p990912.c type testcases, or stuff in directories called misc, I
> would not object.

Yup, yup.

brgds, H-P

Re: Testsuite reorganization, documentation, cleanup?

[Stan Shebs]

Nathanael Nerode wrote:

>Looking at the testsuite, it's rather confusing.  Some directories are
>documented in READMEs; many aren't.  There are multiple organizational
>systems.  There are multiple drivers, many of which seem very similar;
>one suspects redundancy.
>
Since most of the expect scripts run tests on *.c and suchlike (since
99% of the tests don't need special scripts), much of the organization
is forced by the need to have all source files in a directory treated
alike.

There have been rounds of testsuite cleanup in the past, but in some
cases the rationale wasn't written down, and it's not always as obvious
5-10 years later.  For instance, the theory of dates for a test name
was that that the test didn't necessarily have a "theme" or specific
feature that was being tested, so using the date is a simple way to
generate a canonical name.

>
>  Beneath that, a subdivision by correctness, code quality, or compiler
>  behavior quality.
>
Code quality tests are pretty rare, and I don't think we have enough
of them to know how to organize sensibly yet.  I've been thinking
about how to do this kind of test off and on, no doubt there will be
much tcl code involved. :-)

>
>  Beneath that, subdivision by category where appropriate (abi,
>  templates, etc.)
>
>  Beneath that, one directory for each variety (compile, execute, fail to
>  compile, produce warning 'blah')
>
I don't like this level of subdivision very much.  It's needed for
torture tests because they don't have any info internal to the file,
but in general I think test instructions instructions for a file should
be in the file, not implicit in some detail of its pathname.

>2. What do the various existing undocumented directories represent?
>
>For instance, why are there 'gcc.c-torture', 'gcc.dg',
>and 'gcc.misc-tests'?  What are the tests in g++.dg in the subdirectories
>'bprob','debug','gcov','opt',and 'tls', and why aren't they mentioned in
>g++.dg/README ?  
>
Documentation standards for testsuite have been pretty lax.  More
general description would be good, and if at some point, the doc has
to say "this sounds stupid, but has to be this way for hysterical
reasons", that's your clue that a reorg is desirable.

Stan

Re: Testsuite reorganization, documentation, cleanup?

[Nathanael Nerode]

On Mon, Oct 07, 2002 at 02:58:17PM -0700, Stan Shebs wrote:
> Since most of the expect scripts run tests on *.c and suchlike (since
> 99% of the tests don't need special scripts), much of the organization
> is forced by the need to have all source files in a directory treated
> alike.

<snip...>
> I don't like this level of subdivision very much.  It's needed for
> torture tests because they don't have any info internal to the file,
> but in general I think test instructions instructions for a file should
> be in the file, not implicit in some detail of its pathname.
Perhaps torture tests *should* have info internal to the file?

Perhaps we should simply discourage people from adding tests to the
torture suite because of this?

> Documentation standards for testsuite have been pretty lax.  More
> general description would be good, and if at some point, the doc has
> to say "this sounds stupid, but has to be this way for hysterical
> reasons", that's your clue that a reorg is desirable.

Thanks.  I'll use that as the standard!  :-)

--Nathanael

Re: Testsuite reorganization, documentation, cleanup?

[Hans-Peter Nilsson]

On Tue, 8 Oct 2002, Nathanael Nerode wrote:
> On Mon, Oct 07, 2002 at 02:58:17PM -0700, Stan Shebs wrote:
> > I don't like this level of subdivision very much.  It's needed for
> > torture tests because they don't have any info internal to the file,
> > but in general I think test instructions instructions for a file should
> > be in the file, not implicit in some detail of its pathname.

Why not?  What's wrong with having a directory where different
optimization options are iterated over, with the exact option
not explicitly stated in the file?  It's quite convenient IMHO.
If there's a future -O0.5 option, it can be added to the
iterations.

> Perhaps torture tests *should* have info internal to the file?
>
> Perhaps we should simply discourage people from adding tests to the
> torture suite because of this?

There needs to be a replacement, where tests are iterated over
with different (optimization) options.  Currently there is no
gcc.dg (my preferred test-type) directory where that happens.

brgds, H-P

Re: Testsuite reorganization, documentation, cleanup?

[Janis Johnson]

On Tue, Oct 08, 2002 at 03:43:56AM -0400, Hans-Peter Nilsson wrote:
> On Tue, 8 Oct 2002, Nathanael Nerode wrote:
> > On Mon, Oct 07, 2002 at 02:58:17PM -0700, Stan Shebs wrote:
> > > I don't like this level of subdivision very much.  It's needed for
> > > torture tests because they don't have any info internal to the file,
> > > but in general I think test instructions instructions for a file should
> > > be in the file, not implicit in some detail of its pathname.
> 
> Why not?  What's wrong with having a directory where different
> optimization options are iterated over, with the exact option
> not explicitly stated in the file?  It's quite convenient IMHO.
> If there's a future -O0.5 option, it can be added to the
> iterations.

My proposed new test suite for C++ binary compatibility
(http://gcc.gnu.org/ml/gcc-patches/2002-10/msg00329.html) adds a new
test directory under g++.dg and treats all tests in that directory in
a new way.  Information about how a test is run is contained within the
.exp file in that directory rather than in the tests.  The suite also
provides for building each test with a variety of compiler options, with
defaults in the test suite that can be overridden in the environment.

It makes sense to me to keep tests like this separate, with information
about running the tests in the .exp script.  If there were instructions
within the tests, they would be the same for every test, and I don't see
the point of that.

Documentation for these tests is mostly in a new top-level testsuite
README.compat file, with some information also in the new .exp files.
I'd be happy to move or copy it based on the outcome of this discussion.
I can also add documentation about gcov and bprof tests.

Janis

Re: Testsuite reorganization, documentation, cleanup?

[Stan Shebs]

Hans-Peter Nilsson wrote:

>On Tue, 8 Oct 2002, Nathanael Nerode wrote:
>
>>On Mon, Oct 07, 2002 at 02:58:17PM -0700, Stan Shebs wrote:
>>
>>>I don't like this level of subdivision very much.  It's needed for
>>>torture tests because they don't have any info internal to the file,
>>>but in general I think test instructions instructions for a file should
>>>be in the file, not implicit in some detail of its pathname.
>>>
>
>Why not?  What's wrong with having a directory where different
>optimization options are iterated over, with the exact option
>not explicitly stated in the file?  It's quite convenient IMHO.
>If there's a future -O0.5 option, it can be added to the
>iterations.
>
That's why I said "in general".  The current torture test structure
is very efficient for the purpose it serves, and I don't see much
need to change it, although it would be very useful to document it more.

A amusing pragmatic reason: someday the compiler will ICE with -Os on an
empty file. :-)

This touches on a meta-reason for not trying to force every test into a
completely uniform framework; as with monoculture farming, you're
exposed to bugs that only happen in programs not structured in the same
way.  That's why I discouraged uniform GNU-style formatting for the GDB
testsuite programs; it's easy to imagine GDB bugs where source lines
always list correctly for GNU formatting, and incorrectly for other
styles.  (A way to encourage the One True Style I suppose. :-) )

Stan

Re: Testsuite reorganization, documentation, cleanup?

[Nathanael Nerode]

On Tue, Oct 08, 2002 at 09:19:46AM -0700, Stan Shebs wrote:
> Hans-Peter Nilsson wrote:
> 
> >On Tue, 8 Oct 2002, Nathanael Nerode wrote:
> >
> >>On Mon, Oct 07, 2002 at 02:58:17PM -0700, Stan Shebs wrote:
> >>
> >>>I don't like this level of subdivision very much.  It's needed for
> >>>torture tests because they don't have any info internal to the file,
> >>>but in general I think test instructions instructions for a file should
> >>>be in the file, not implicit in some detail of its pathname.
> >>>
> >
> >Why not?  What's wrong with having a directory where different
> >optimization options are iterated over, with the exact option
> >not explicitly stated in the file?  It's quite convenient IMHO.
> >If there's a future -O0.5 option, it can be added to the
> >iterations.
> >
> That's why I said "in general".  The current torture test structure
> is very efficient for the purpose it serves, and I don't see much
> need to change it, although it would be very useful to document it more.

...soooo, how about someone approving my first doc cleanup patch (v3)
http://gcc.gnu.org/ml/gcc-patches/2002-10/msg00303.html

...or explaining why it's no good?

--Nathanael

Re: Testsuite reorganization, documentation, cleanup?

[Janis Johnson]

On Wed, Oct 09, 2002 at 12:48:12PM -0400, Nathanael Nerode wrote:
> 
> ...soooo, how about someone approving my first doc cleanup patch (v3)
> http://gcc.gnu.org/ml/gcc-patches/2002-10/msg00303.html
> 
> ...or explaining why it's no good?

This doc cleanup patch looks good to me, although I don't have the
authority to approve it.

Janis

Re: Testsuite reorganization, documentation, cleanup?

[Ben Elliston]

>>>>> "Nathanael" == Nathanael Nerode <neroden@doctormoo.dyndns.org> writes:

  Nathanael> 1. What do people think the ideal organization should be?

One thing to add: my local tree has a `gcc.target' subdirectory (with
subdirectories named after target triplets) that contain
target-specific tests.  When the time came to write some earlier this
year, I realised there was nothing in place and so I devised the
cleanest layout I could.

The FSF GCC tree should probably get a gcc.target subdirectory as
well.

Ben