testsuite layout conventions

testsuite layout conventions

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 3012 bytes --]

do we have docs/guidelines for testsuite/ dir layouts ?  we seem to
(inconsistently) use a lot of redundant layouts largely due to our
use of dejagnu.

dejagnu takes a tool-centric view of the world.  this makes sense
when you have a flat testsuite dir (e.g. if gas & binutils & ld &
gdb all shared a testsuite/ dir), but i'm not sure it makes that
much sense when they're all sep.  i think dejagnu is the single
reason for the layouts we have and it seems a bit unnecessary.

when you set tool=<tool> in its site.exp, it will only search for
dirs with <tool> prefixes under testsuite/.  i.e. it applies the
glob like testsuite/<tool>*/**.

a quick survey of binutils/gdb/gcc dirs that have tests:
* binutils     / testsuite / binutils-<group>  / <tests>
* gas          / testsuite / gas / <group>     / <tests>
* gdb          / testsuite / gdb.<group>       / <tests>
* gold         / testsuite / <tests>
* ld           / testsuite / ld-<group>        / <tests>
* libctf       / testsuite / libctf-<group>    / <tests>
* libiberty    / testsuite / <tests>
* sim          / testsuite / sim / <group>     / <tests>
* gcc          / testsuite / <tool>.<group>    / <tests>
* libatomic    / testsuite / libatomic.<group> / <tests>
* libffi       / testsuite / libffi.<group>    / <tests>
* libgomp      / testsuite / libgomp.<group>   / <tests>
* libgo        / testsuite / libgo.<group>     / <tests>
* libitm       / testsuite / libitm.<group>    / <tests>
* libphobos    / testsuite / libphobos.<group> / <tests>
* libstdc++-v3 / testsuite / libstdc++-<group> / <tests>
* libvtv       / testsuite / libvtv.<group>    / <tests>

the notable standouts are gold & libiberty.  they don't use dejagnu
while everyone else does.  so i'll ignore them.

libstdc++-v3 has additional test subdirs beyond the tool=libstdc++
prefix.  not sure how that works exactly.  so i'll ignore them.

gcc is also a bit funky, but i guess that's what you get for such a
large subdir.  i don't think we want to use that as precedence :).

for all the rest, you can see that we have three styles:
	<tool>.<group>/
	<tool>-<group>/
	<tool>/<group>/

the odd thing is that it seems most projects don't actually set
tool in their site.exp.  libstdc++, gdb, and sim do.  so i guess
either the others ones used to, or they just adopted a similar
naming convention at some point because others were ?

i've just finished cleaning up the sim/testsuite/ tree.  i'm left
with sim/testsuite/sim/<arch>/<tests>.  i spend a lot of time in
sim/<arch>/ and sim/testsuite/sim/<arch>/ comparing and trying to
collapse common logic.  the extra sim/ dir irks me ;).  it ends up
being easy to collapse by deleting 'set tool sim' from site.exp,
then just doing:
  cd testsuite/
  git mv sim/* ./
which leaves me with sim/testsuite/<arch>/.

based on the fact that no one seems to use dejagnu's tool setting
anymore, and we don't seem to have general guidance, and every
project is slightly different, changing sim to drop the "sim" tool
prefix is fine.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: testsuite layout conventions

[Andreas Schwab]

On Jan 06 2021, Mike Frysinger via Binutils wrote:

> the odd thing is that it seems most projects don't actually set
> tool in their site.exp.  libstdc++, gdb, and sim do.  so i guess
> either the others ones used to, or they just adopted a similar
> naming convention at some point because others were ?

It can also be set on the command line.  This is used by automake.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: testsuite layout conventions

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

On 06 Jan 2021 09:33, Andreas Schwab wrote:
> On Jan 06 2021, Mike Frysinger via Binutils wrote:
> > the odd thing is that it seems most projects don't actually set
> > tool in their site.exp.  libstdc++, gdb, and sim do.  so i guess
> > either the others ones used to, or they just adopted a similar
> > naming convention at some point because others were ?
> 
> It can also be set on the command line.  This is used by automake.

ah, thanks, i missed the projects using newer automake & DEJATOOL and
passing --tool to runtest directly rather than through site.exp.  i
had expected site.exp so people could run `runtest` manually, but i
guess that's not really a common or supported flow.  people only run
`make check` and such and never get lower.

i'm still stuck in the old Makefile.in world -- it still feels cutting
edge that binutils/gdb is using Makefile.am at all as i thought it'd
never happen ;).
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: testsuite layout conventions

[Alan Modra]

On Wed, Jan 06, 2021 at 09:33:24AM +0100, Andreas Schwab wrote:
> On Jan 06 2021, Mike Frysinger via Binutils wrote:
> 
> > the odd thing is that it seems most projects don't actually set
> > tool in their site.exp.  libstdc++, gdb, and sim do.  so i guess
> > either the others ones used to, or they just adopted a similar
> > naming convention at some point because others were ?
> 
> It can also be set on the command line.  This is used by automake.

Yes, we use that in binutils, gas and ld.  I once tried to flatten out
the gas testsuite heirarchy, removing --tool from the runtest
invocation.  That ran into trouble with gas-dg.exp I think it was,
with -dg-test being called instead of gas-dg-test.

-- 
Alan Modra
Australia Development Lab, IBM