testsuite/run_dump_test improvement

testsuite/run_dump_test improvement

[Fangrui Song]

Hi all, may the force be with you!

I have known check-gas, check-ld and check-binutils for a while, as well
as "many tests are organized as these *.d files", but I know very little
about the underlying test framework. TIL
binutils/testsuite/lib/binutils-common.exp run_dump_test is the core
that runs as, ld, and checks the output of a dump program matches the
expected patterns in the .d file (or #dump:).

Here is my first complaint: the test framework is undocumented.

2) Can we write comments in *.d?

3) Can we allow free-form shell commands instead of specialized directives?

It is mythical what can and what can't be used in `#foo:`.
OK, a user can figure out this with trial and error. Then,
why can't we have multiple dump programs? Frequently we need to check
multiple properties of an output and a single dump program just does not
work well.  The limitation led to some ad-hoc directives like `#map:`.
For example, ld-ifunc/ifunc-1-x86.d says

   #ld: -shared -Map tmpdir/ifunc-1-x86.map --hash-style=sysv
   #objdump: -dw
   #target: x86_64-*-* i?86-*-*
   #map: ifunc-1-x86.map
   
   #...
   [ \t0-9a-f]+:[ \t0-9a-f]+call[ \t0-9a-fq]+<\*ABS\*(\+0x[0-9a-f]+|)@plt>
   #pass

If we allow free-form shell commands and use line prefixes to differentiate the two output streams (objdump -d output and ld -Map output):

   ## Test local ifunc are dumped in the link map.
   #RUN: ld: -shared -Map tmpdir/ifunc-1-x86.map --hash-style=sysv %s -o %t
   #RUN: objdump -dw %t | check  # by default, CHECK is the prefix
   #RUN: cat tmpdir/ifunc-1-x86.map | check MAP

   #CHECK: [ \t0-9a-f]+:[ \t0-9a-f]+call[ \t0-9a-fq]+<\*ABS\*(\+0x[0-9a-f]+|)@plt>
   #CHECK-N: if there is a next line
   #CHECK-N: likewise.
   #CHECK:   skip arbitrary lines
   #CHECK-N: if there is a next line

   #MAP: Local IFUNC function ...

4) It may be nice for the pattern matching tool to recognize a substring by default.
Let {{^}}, {{$}}, or some other metacharacters to denote that "the full line should be matched"

5) It may be nice to use more descriptive filenames than *-1.d, *-2.d, or prXXXXX.d
   prXXXXX can be mentioned as comments if we allow 2)

   As it stands, many tests require guesswork before you open them in an editor.

6) Parallelism

   runtest --tool ld --srcdir ../../ld/testsuite/ took more than 2 hours
   to run on my machine. I'd expect a few seconds for a more pleasant experience.

7) Run an individual test, rather than runtest ...

I figured out that I can run partial tests with:

   # my build directory is $srcdir/Debug
   cd Debug/ld
   runtest --tool ld --srcdir ../../ld/testsuite export-dynamic-symbol.exp

but it would be convenient to be able to run just one test.


Cheers

Re: testsuite/run_dump_test improvement

[Hans-Peter Nilsson]

On Mon, 4 May 2020, Fangrui Song via Binutils wrote:

> Hi all, may the force be with you!
>
> I have known check-gas, check-ld and check-binutils for a while, as well
> as "many tests are organized as these *.d files", but I know very little
> about the underlying test framework. TIL
> binutils/testsuite/lib/binutils-common.exp run_dump_test is the core
> that runs as, ld, and checks the output of a dump program matches the
> expected patterns in the .d file (or #dump:).
>
> Here is my first complaint: the test framework is undocumented.

Hey, it's documented in the source, and since you found the
file, that can't be too bad and you should have found the answer
to those of your questions.  If you want more, patches are
likely welcome.  (Having said that, if you don't derive the
documentation from the source, it's bound to be outdated.)

brgds, H-P

Re: testsuite/run_dump_test improvement

[Nick Clifton]

Hi Fangrui,

> 2) Can we write comments in *.d?

Yes.  There are plenty of example of this.  Just prefix them with a # character.
For an example see gas/testsuite/gas/elf/bignums.d


> 3) Can we allow free-form shell commands instead of specialized directives?

No - the *.d files are not shell scripts.  They are Tcl command files,

> Then, why can't we have multiple dump programs?

We can - we just need to write a proc to support such behaviour.  Or extend
an existing proc.

> Frequently we need to check
> multiple properties of an output and a single dump program just does not
> work well. 

But you can have multiple *.d files that reference the same source file
and which use different dump tools to check different properties of the
output.  Maybe not as elegant as having all the checks in one file, but
certainly do-able with the current infrastructure.


> 4) It may be nice for the pattern matching tool to recognize a substring by default.
> Let {{^}}, {{$}}, or some other metacharacters to denote that "the full line should be matched"

I am not clear as to what you mean here.  If you need an exact match of
a line of output then you just include those characters as a line in the
.d file.  eg:

  must match this line exactly

If you want to insist that a line is present but you do not care about
the characters on the line then you can just you a glob-regexp, eg:

  .*

If you want to match multiple lines up until a specific line then you
can use #...  For example:

  must match this line
  #...
  any number of lines were matched until this line, but this line must be present

Plus if you no longer care about the rest of the output and you just want
the test to pass you can use:

  #pass


> 5) It may be nice to use more descriptive filenames than *-1.d, *-2.d, or prXXXXX.d
>   prXXXXX can be mentioned as comments if we allow 2)
>   As it stands, many tests require guesswork before you open them in an editor.

True - but this is down to the testcase writers.  Or a maintainer to 
come along and rename them....


> 6) Parallelism
> 
>   runtest --tool ld --srcdir ../../ld/testsuite/ took more than 2 hours
>   to run on my machine. I'd expect a few seconds for a more pleasant experience.

I think that the gcc testsuite has the concept of expensive tests which can
be disabled via an environment option.  Maybe we need something similar for
the binutils testsuites.


> 7) Run an individual test, rather than runtest ...
> 
> I figured out that I can run partial tests with:
> 
>   # my build directory is $srcdir/Debug
>   cd Debug/ld
>   runtest --tool ld --srcdir ../../ld/testsuite export-dynamic-symbol.exp

Which can also be done as:

    make check RUNTESTFLAGS="<dir>/<driver>"
eg:
    make check-ld RUNTESTFLAGS="ld-elf/elf.exp"

> but it would be convenient to be able to run just one test.

When I need to do this I just create a dummy test driver file
that only runs the test I want.  eg:

  % cat ld/testsuite/ld-elf/dummy.exp
  run_dump_test seg

  % make check-ld RUNTESTFLAGS="ld-elf/dummy.exp"
  [...]
  Running /work/sources/binutils/current/ld/testsuite/ld-elf/dummy.exp ...

		=== ld Summary ===

  # of expected passes		1

It is ungainly I agree, but it does work.

Cheers
  Nick

Re: testsuite/run_dump_test improvement

[Fangrui Song]

On Wed, May 6, 2020 at 6:44 AM Nick Clifton via Binutils
<binutils@sourceware.org> wrote:
>
> Hi Fangrui,
>
> > 2) Can we write comments in *.d?
>
> Yes.  There are plenty of example of this.  Just prefix them with a # character.
> For an example see gas/testsuite/gas/elf/bignums.d
>
>
> > 3) Can we allow free-form shell commands instead of specialized directives?
>
> No - the *.d files are not shell scripts.  They are Tcl command files,
>
> > Then, why can't we have multiple dump programs?
>
> We can - we just need to write a proc to support such behaviour.  Or extend
> an existing proc.
>
> > Frequently we need to check
> > multiple properties of an output and a single dump program just does not
> > work well.
>
> But you can have multiple *.d files that reference the same source file
> and which use different dump tools to check different properties of the
> output.  Maybe not as elegant as having all the checks in one file, but
> certainly do-able with the current infrastructure.
>
>
> > 4) It may be nice for the pattern matching tool to recognize a substring by default.
> > Let {{^}}, {{$}}, or some other metacharacters to denote that "the full line should be matched"
>
> I am not clear as to what you mean here.  If you need an exact match of
> a line of output then you just include those characters as a line in the
> .d file.  eg:
>
>   must match this line exactly
>
> If you want to insist that a line is present but you do not care about
> the characters on the line then you can just you a glob-regexp, eg:
>
>   .*
>
> If you want to match multiple lines up until a specific line then you
> can use #...  For example:
>
>   must match this line
>   #...
>   any number of lines were matched until this line, but this line must be present
>
> Plus if you no longer care about the rest of the output and you just want
> the test to pass you can use:
>
>   #pass
>
>
> > 5) It may be nice to use more descriptive filenames than *-1.d, *-2.d, or prXXXXX.d
> >   prXXXXX can be mentioned as comments if we allow 2)
> >   As it stands, many tests require guesswork before you open them in an editor.
>
> True - but this is down to the testcase writers.  Or a maintainer to
> come along and rename them....
>
>
> > 6) Parallelism
> >
> >   runtest --tool ld --srcdir ../../ld/testsuite/ took more than 2 hours
> >   to run on my machine. I'd expect a few seconds for a more pleasant experience.
>
> I think that the gcc testsuite has the concept of expensive tests which can
> be disabled via an environment option.  Maybe we need something similar for
> the binutils testsuites.
>
>
> > 7) Run an individual test, rather than runtest ...
> >
> > I figured out that I can run partial tests with:
> >
> >   # my build directory is $srcdir/Debug
> >   cd Debug/ld
> >   runtest --tool ld --srcdir ../../ld/testsuite export-dynamic-symbol.exp
>
> Which can also be done as:
>
>     make check RUNTESTFLAGS="<dir>/<driver>"
> eg:
>     make check-ld RUNTESTFLAGS="ld-elf/elf.exp"
>
> > but it would be convenient to be able to run just one test.
>
> When I need to do this I just create a dummy test driver file
> that only runs the test I want.  eg:
>
>   % cat ld/testsuite/ld-elf/dummy.exp
>   run_dump_test seg
>
>   % make check-ld RUNTESTFLAGS="ld-elf/dummy.exp"
>   [...]
>   Running /work/sources/binutils/current/ld/testsuite/ld-elf/dummy.exp ...
>
>                 === ld Summary ===
>
>   # of expected passes          1
>
> It is ungainly I agree, but it does work.
>
> Cheers
>   Nick
>

I filed https://sourceware.org/bugzilla/show_bug.cgi?id=28602
("binutils/testsuite/lib/binutils-common.exp: Support free-form shell
commands and check patterns").

I think some new RISC-V contributors have found pain in existing
testsuite and when adding new tests.
Perhaps they may take a stab...