[patch, gas, documentation] Some additional testsuite info

[patch, gas, documentation] Some additional testsuite info

[Thomas Koenig]

[-- Attachment #1: Type: text/plain, Size: 365 bytes --]

Hi,

here's some additional info on the gas testsuite.  I also took the
liberty to delete the "kind of lame" comment, which is not really
appropriate any more, at least not for the architectures which
were added in the last few decades.

It was actually quite amusing to run the testsuite on an architecture
which did not (yet) have a NOP :-)

Best regards

	Thomas

[-- Attachment #2: testsuite_doc.txt --]
[-- Type: text/plain, Size: 2898 bytes --]

diff --git a/gas/doc/internals.texi b/gas/doc/internals.texi
index 56f10fd16fa..b2d2e0f39b8 100644
--- a/gas/doc/internals.texi
+++ b/gas/doc/internals.texi
@@ -1154,7 +1154,7 @@ types not mentioned here.
 The second parameter is @var{litP} which is a pointer to a byte array where the
 converted value should be stored.  The value is converted into LITTLENUMs and
 is stored in the target's endian-ness order.  (@var{LITTLENUM} is defined in
-gas/bignum.h).  Single precision values occupy 2 littlenums.  Double precision
+@file{gas/bignum.h}).  Single precision values occupy 2 littlenums.  Double precision
 values occupy 4 littlenums and extended precision values occupy either 5 or 6
 littlenums, depending upon the target.
 
@@ -1934,10 +1934,20 @@ original filename and line number that they are applicable to.
 @section Test suite
 @cindex test suite
 
-The test suite is kind of lame for most processors.  Often it only checks to
-see if a couple of files can be assembled without the assembler reporting any
-errors.  For more complete testing, write a test which either examines the
-assembler listing, or runs @code{objdump} and examines its output.  For the
+The test suite uses dejagnu.  It is invoked by running @command{make check} from
+the main gas directory or, recursively, from the main binutils direcctory.
+
+Some tests are run for all targets. These are situated in the
+@file{gas/testsuite/gas/all} subdirectory. These tests require a @code{nop}
+instruction, otherwise many of them will fail, including crashing the test
+suite driver.
+
+If you create a subdirectory for architecture foo_arch, create a file
+@file{foo_arch/foo_arch.exp} which contains the TCL commands run by
+dejagnu.
+
+It is best to write tests which either examine the
+assembler listing, or run @code{objdump} and examine its output.  For the
 latter, the TCL procedure @code{run_dump_test} may come in handy.  It takes the
 base name of a file, and looks for @file{@var{file}.d}.  This file should
 contain as its initial lines a set of variable settings in @samp{#} comments,
@@ -1959,10 +1969,17 @@ The non-commented parts of the file are interpreted as regular expressions, one
 per line.  Blank lines in the @code{objdump} or @code{nm} output are skipped,
 as are blank lines in the @code{.d} file; the other lines are tested to see if
 the regular expression matches the program output.  If it does not, the test
-fails.
+fails.  Whitespace is significant.  Note that this means the tests must be
+modified if the @code{objdump} output style is changed.
+
+A simple example of this is contained in the @file{testsuite/gas/aarch64}
+directory.
 
-Note that this means the tests must be modified if the @code{objdump} output
-style is changed.
+If you want to run only the tests for foo_arch, use
+
+@example
+$ make check RUNTESTFLAGS="foo_arch.exp"
+@end example
 
 @bye
 @c Local Variables:

Re: [patch, gas, documentation] Some additional testsuite info

[Jan Beulich]

On 30.04.2023 12:16, Thomas Koenig via Binutils wrote:
> here's some additional info on the gas testsuite.  I also took the
> liberty to delete the "kind of lame" comment, which is not really
> appropriate any more, at least not for the architectures which
> were added in the last few decades.

Up-front remark: Non-inlined patches are hard to comment upon.

You refer to "the main binutils directory", which is ambiguous next
to "the main gas directory". Itym the top-level directory there
(and not e.g. the binutils/ one being a sibling of gas/).

The reference to "a @code{nop} instruction" is ambiguous, too, I'm
afraid: There's no such requirement (anymore) if you mean to refer
to an instruction truly named "nop" and not taking any operands.
Many architectures don't have such. But this has been overcome by
the introduction of the .nop directive, which merely requires the
arch to have some insn which is "no operation" (which might e.g.
be ORin zero into a register). I'm pretty sure almost all
architectures will have such. I'm also quite sure though that there
are old tests which are yet to be converted ...

You move but otherwise retain the sentence regarding the need to
change expectations when objdump output style changes. I think
this should be relaxed at least as slightly as saying "may need to
be" instead of "must". Generally you'll find that more modern tests
write expectations in suitably relaxed regexp-s such that quite a
few possible layout/style changes wouldn't require adjustments. And
I think if doc is adjusted in this area anyway, it should recommend
to write expectations as tight as necessary (for the purpose of
the test), but as relaxed as possible.

Jan