Testsuite for avr-as

Testsuite for avr-as

[Erik Christiansen]

G'day,

Unlike other gas implementations, avr-as lacks a formal testsuite. Over
on the avr-gcc list, a couple of us are seeking to rectify that. Rather
than go too far with "home-brew" test schemes, we're trying to scrabble
up the binutils testsuite learning curve.

A bit of poking around in the binutils source distribution shows .s and
.d files. Is there some doco that we can read, to discover:

a) Preferred method for generating .d files.
b) Rather than further develop our existing prototype test engines, what
   script does binutils use to bang through the .s/.d pairs?

Oh, a horrible thought: If this is to find its way into the distribution
one day, then there's likely to be some requirements we should know about.

Erik

Re: Testsuite for avr-as

[Hans-Peter Nilsson]

On Tue, 7 Dec 2004, Erik Christiansen wrote:

> G'day,
>
> Unlike other gas implementations, avr-as lacks a formal testsuite. Over
> on the avr-gcc list, a couple of us are seeking to rectify that. Rather
> than go too far with "home-brew" test schemes, we're trying to scrabble
> up the binutils testsuite learning curve.
>
> A bit of poking around in the binutils source distribution shows .s and
> .d files. Is there some doco that we can read, to discover:
>
> a) Preferred method for generating .d files.

Method?  Hah!  The output of either objdump or readelf
redirected, then inspected for sanity and edited to regexpify
the contents.

> b) Rather than further develop our existing prototype test engines, what
>    script does binutils use to bang through the .s/.d pairs?

Something tells me you're not aware how the DejaGNU-based
testsuite (usually) works.  Build binutils, then run "make -k
check" and watch DejaGnu load and execute the various .exp files
and run whatever is said there, usually using stuff from the
installed DejaGnu and gas/testsuite/gas/lib.  (You can do this
for AVR if you want; some tests are generic.)

My favorite is run_dump_test iterating over .d files.  See
gas/testsuite/gas/mmix/mmix.exp.

Sometimes you need to test for proper error message or error
handling.  See gas/testsuite/gas/mmix/mmix-err.exp.  (No .d,
just source files.)

Sometimes you want to just check that the assembly listing looks
about right.  See gas/testsuite/gas/mmix/mmix-list.exp.
Usually the test iterates over ".l" files.

> Oh, a horrible thought: If this is to find its way into the distribution
> one day, then there's likely to be some requirements we should know about.

Good thinking.  Copyright assignment *papers* signed from *all*
contributors. (Hopefully Nick Clifton, head binutils maintainer,
will fill in here with proper references.)

brgds, H-P

Re: Testsuite for avr-as

[Erik Christiansen]

On Mon, Dec 06, 2004 at 08:11:45PM -0500, Hans-Peter Nilsson wrote:
> On Tue, 7 Dec 2004, Erik Christiansen wrote:
> > a) Preferred method for generating .d files.
> 
> Method?  Hah!  The output of either objdump or readelf
> redirected, then inspected for sanity and edited to regexpify
> the contents.

Your one sentence is more valuable than the whole DejaGNU manual, at the
moment. Digesting that edifice, in order merely to comprehend the
permissible .d grammar, is a steep climb.

We were wondering how best to handle permissible variability in the
output. You've answered that question too!   :-)

> > b) Rather than further develop our existing prototype test engines, what
> >    script does binutils use to bang through the .s/.d pairs?
> 
> Something tells me you're not aware how the DejaGNU-based
> testsuite (usually) works.  Build binutils, then run "make -k
> check" and watch DejaGnu load and execute the various .exp files
> and run whatever is said there, usually using stuff from the
> installed DejaGnu and gas/testsuite/gas/lib.  (You can do this
> for AVR if you want; some tests are generic.)

I tried running "make -k check" in my obj-avr/, to see if anything
interesting happened. It was kind enough to create a new site.exp file.
Now I just need to figure out how to connect that to our .s/.d pairs,
rather than hare off after " build_triplet i686-pc-linux-gnu", etc.
(A bit of RTFM should help, in time.)

But for a start, your simplest case may suffice:

[snip]

> Sometimes you want to just check that the assembly listing looks
> about right.  See gas/testsuite/gas/mmix/mmix-list.exp.
> Usually the test iterates over ".l" files.

Some quick results, using DejaGnu-compatible .s/.d pairs, would be very
motivating, so I'm even tempted to refresh my TCL acquaintance, and try
just running them through "regexp_diff", once I figure out whether it's
a built-in, or we have to borrow it from one of the scripts.

To bash through a slab of .s/.d pairs for an assembler, which should
give very consistent output, regexp_diff and our 8-line bash script
would do the job, I suspect. Once some more test cases are happening,
the layers of DejaGnu flexibility-providing obfuscation could then be a
second step.

Thanks very much for your help!

Erik

Re: Testsuite for avr-as

[Hans-Peter Nilsson]

On Tue, 7 Dec 2004, Erik Christiansen wrote:
> I tried running "make -k check" in my obj-avr/, to see if anything
> interesting happened. It was kind enough to create a new site.exp file.

You should also have got test results from the generic parts of
the test-suites, and output in obj-avr/gas/testsuite/gas.log.

> Now I just need to figure out how to connect that to our .s/.d pairs,
> rather than hare off after " build_triplet i686-pc-linux-gnu", etc.

Ignore site.exp, it's just there to confuse you.  No magic. Copy
the .exp:s I quoted and just s/mmix/avr/.  All framework set, no
need to copy anything else; go write the actual tests.

> Some quick results, using DejaGnu-compatible .s/.d pairs, would be very
> motivating, so I'm even tempted to refresh my TCL acquaintance, and try
> just running them through "regexp_diff", once I figure out whether it's
> a built-in, or we have to borrow it from one of the scripts.

Heh, actually looking at the source would help...  or not.
It's in gas/testsuite/lib/gas-defs.exp.  (Actually copied all
over; it's in other generic testsuites as well, just like
run_dump_test.)

> To bash through a slab of .s/.d pairs for an assembler, which should
> give very consistent output, regexp_diff and our 8-line bash script
> would do the job, I suspect. Once some more test cases are happening,
> the layers of DejaGnu flexibility-providing obfuscation could then be a
> second step.

You probably want to improve the ld testsuite for avr as well.
Mostly the same thing, just in the ld directory.

brgds, H-P

Re: Testsuite for avr-as

[Hans-Peter Nilsson]

On Tue, 7 Dec 2004, Hans-Peter Nilsson wrote:
> Heh, actually looking at the source would help...  or not.

Oh, and you might actually be helped by the comment above
run_dump_test in gas/testsuite/lib/gas-defs.exp.  It's all the
documentation there is about how to write the
"#directive: parameters" lines at the top of the .d files.

Other than that, intense staring at the existing target and
generic testsuites helps by example.

brgds, H-P

Re: Testsuite for avr-as

[Erik Christiansen]

On Tue, Dec 07, 2004 at 08:06:34AM -0500, Hans-Peter Nilsson wrote:
> On Tue, 7 Dec 2004, Erik Christiansen wrote:
> > Some quick results, using DejaGnu-compatible .s/.d pairs, would be very
> > motivating, so I'm even tempted to refresh my TCL acquaintance, and try
> > just running them through "regexp_diff", once I figure out whether it's
> > a built-in, or we have to borrow it from one of the scripts.
> 
> Heh, actually looking at the source would help...  or not.

It did. :-)

> It's in gas/testsuite/lib/gas-defs.exp.  (Actually copied all
> over; it's in other generic testsuites as well, just like
> run_dump_test.)

Thanks! Found it (after posting, of course.), together with
run_dump_test, which you had advocated. As hinted, it looks like just
the ticket.

> > To bash through a slab of .s/.d pairs for an assembler, which should
> > give very consistent output, regexp_diff and our 8-line bash script
> > would do the job, I suspect. Once some more test cases are happening,
> > the layers of DejaGnu flexibility-providing obfuscation could then be a
> > second step.
> 
> You probably want to improve the ld testsuite for avr as well.
> Mostly the same thing, just in the ld directory.

Yeah, once we've got me up to (a good plodding) speed, better make the
most of it. ;-)

I'll digest your other very helpful advice, and have another go at
eliciting minimal ticking over from the dejagnu machinery.

Thanks again,
Erik

Re: Testsuite for avr-as

[Erik Christiansen]

On Tue, Dec 07, 2004 at 08:06:34AM -0500, Hans-Peter Nilsson wrote:
> Ignore site.exp, it's just there to confuse you.  No magic. Copy
> the .exp:s I quoted and just s/mmix/avr/.  All framework set, no
> need to copy anything else; go write the actual tests.

Maybe not magic, but that was truly the map through the minefield.
After rebuilding binutils, to detect the new testsuite directory, we
have:

Running /usr/local/src/binutils-041119/gas/testsuite/gas/arm/arm.exp ...
Running /usr/local/src/binutils-041119/gas/testsuite/gas/avr/avr.exp ...
Running /usr/local/src/binutils-041119/gas/testsuite/gas/cfi/cfi.exp ...

That leaves two next steps:   (Jump in with both feet? :-)
a) Figure out how to specify the assembler path, so we can regression
   test a newly built one. (Doesn't seem to be an option in the *.d
   head)

b) Figure out why the test run gobbles up 250 ptys, then resumes after a
2 minute wait:

Running /usr/local/src/binutils-041119/gas/testsuite/gas/all/gas.exp ...
ERROR: Can't run gas/all/comment.s: The system has no more ptys.  Ask
your system administrator to create more.
ERROR: timeout

I was hoping to leave Dmitry with a functioning basic environment, for
addition of test cases, before going away for Christmas, at the end of
this week, so any hints on this next level would be gold.

Thanks for your patience,
Erik

Re: Testsuite for avr-as

[Hans-Peter Nilsson]

On Thu, 9 Dec 2004, Erik Christiansen wrote:
> On Tue, Dec 07, 2004 at 08:06:34AM -0500, Hans-Peter Nilsson wrote:
> > Ignore site.exp, it's just there to confuse you.  No magic. Copy
> > the .exp:s I quoted and just s/mmix/avr/.  All framework set, no
> > need to copy anything else; go write the actual tests.
>
> Maybe not magic, but that was truly the map through the minefield.

Now, really.  I don't see how it could be much simpler, save
for not writing any files at all.

> After rebuilding binutils, to detect the new testsuite directory, we
> have:
>
> Running /usr/local/src/binutils-041119/gas/testsuite/gas/arm/arm.exp ...
> Running /usr/local/src/binutils-041119/gas/testsuite/gas/avr/avr.exp ...
> Running /usr/local/src/binutils-041119/gas/testsuite/gas/cfi/cfi.exp ...

Do I understand correctly that you have tests in
gas/testsuite/gas/avr but that they aren't run?

Doing "make check 'RUNTESTFLAGS=-v -v'" should give you more
info, but I think that's mentioned in the dejagnu manual, so you
should already be aware of that step...

This is a good time to read all manuals and have a good look at
the source code.  Lots of info is in the installed dejagnu, at
/usr or /usr/local share/dejagnu

> That leaves two next steps:   (Jump in with both feet? :-)

How else do you expect to swim?  Really.

> a) Figure out how to specify the assembler path, so we can regression
>    test a newly built one. (Doesn't seem to be an option in the *.d
>    head)

No, you don't have to do this, the framework does it for you.
What I said should be all that's needed.  Did you change all
mmix-* to avr-*?

> b) Figure out why the test run gobbles up 250 ptys, then resumes after a
> 2 minute wait:
>
> Running /usr/local/src/binutils-041119/gas/testsuite/gas/all/gas.exp ...
> ERROR: Can't run gas/all/comment.s: The system has no more ptys.  Ask
> your system administrator to create more.
> ERROR: timeout

I have no idea.  Using an old tcl or expect version?

Usually people build in a directory other than where the source
is.  Maybe you don't do that and there are issues with running
the testsuite in the source directory.

brgds, H-P

Re: Testsuite for avr-as

[Hans-Peter Nilsson]

On Thu, 9 Dec 2004, Hans-Peter Nilsson wrote:

> On Thu, 9 Dec 2004, Erik Christiansen wrote:
> > On Tue, Dec 07, 2004 at 08:06:34AM -0500, Hans-Peter Nilsson wrote:
> > > Ignore site.exp, it's just there to confuse you.  No magic. Copy
> > > the .exp:s I quoted and just s/mmix/avr/.  All framework set, no
> > > need to copy anything else; go write the actual tests.
> >
> > Maybe not magic, but that was truly the map through the minefield.
>
> Now, really.  I don't see how it could be much simpler, save
> for not writing any files at all.

I guess I misinterpreted what you meant by that.  (Still,
copying the framework from another target seems pretty obvious.)

> > b) Figure out why the test run gobbles up 250 ptys, then resumes after a
> > 2 minute wait:
> >
> > Running /usr/local/src/binutils-041119/gas/testsuite/gas/all/gas.exp ...
> > ERROR: Can't run gas/all/comment.s: The system has no more ptys.  Ask
> > your system administrator to create more.
> > ERROR: timeout
>
> I have no idea.  Using an old tcl or expect version?

One way to sanity-check your dejagnu+expect+tcl installation
would be to build and make check for your host system (that is,
not specifying "--target=...".  There should be no test failures
there.

brgds, H-P

Re: Testsuite for avr-as

[Erik Christiansen]

On Thu, Dec 09, 2004 at 06:20:31AM -0500, Hans-Peter Nilsson wrote:
> On Thu, 9 Dec 2004, Erik Christiansen wrote:
> > On Tue, Dec 07, 2004 at 08:06:34AM -0500, Hans-Peter Nilsson wrote:
> > > Ignore site.exp, it's just there to confuse you.  No magic. Copy
> > > the .exp:s I quoted and just s/mmix/avr/.  All framework set, no
> > > need to copy anything else; go write the actual tests.
> >
> > Maybe not magic, but that was truly the map through the minefield.
> 
> Now, really.  I don't see how it could be much simpler, save
> for not writing any files at all.

Oh-Oh!! I've offended with my praise of help received! Sorry. :-(

It is the benefit of your knowledge which has made it simpler for me too.
Thankyou! :-)

> > After rebuilding binutils, to detect the new testsuite directory, we
> > have:
> >
> > Running /usr/local/src/binutils-041119/gas/testsuite/gas/arm/arm.exp ...
> > Running /usr/local/src/binutils-041119/gas/testsuite/gas/avr/avr.exp ...
> > Running /usr/local/src/binutils-041119/gas/testsuite/gas/cfi/cfi.exp ...
> 
> Do I understand correctly that you have tests in
> gas/testsuite/gas/avr but that they aren't run?

Nope, my message was intended to be positive, but here I failed to make
clear that the tests are running. I was, and am, thrilled with the
progress. :-)

In fact, the informative error messages shown in the log erased any
doubts that dejagnu is the way to go. 

[snip]

> > a) Figure out how to specify the assembler path, so we can regression
> >    test a newly built one. (Doesn't seem to be an option in the *.d
> >    head)
> 
> No, you don't have to do this, the framework does it for you.
> What I said should be all that's needed.  Did you change all
> mmix-* to avr-*?

Nope, I followed the Dec 07 advice, quoted at the top of this email,
copying only mmix.exp to gas/testsuite/gas/avr/avr.exp, and performing
the substitution. Was that the right thing to do?

Would you be able to hint how "the framework does it for you"?
Looking in the Makefile in obj-avr/gas, to find out where config is
picked up, the "check" target becomes "check-recursive", which doesn't
seem to be there. So now I'm digging through run_dump_test, since it
must be revealed that way. (Eventually :-)

> > b) Figure out why the test run gobbles up 250 ptys, then resumes after a
> > 2 minute wait:
> >
> > Running /usr/local/src/binutils-041119/gas/testsuite/gas/all/gas.exp ...
> > ERROR: Can't run gas/all/comment.s: The system has no more ptys.  Ask
> > your system administrator to create more.
> > ERROR: timeout
> 
> I have no idea.  Using an old tcl or expect version?

Thanks for the tip! I'll try upgrading related debian woody packages.

> Usually people build in a directory other than where the source
> is.  Maybe you don't do that and there are issues with running
> the testsuite in the source directory.

No way! Though I had been getting away with that for several years,
there has been enough traffic on this topic, on several lists,
indicating that it is not supported.

Thanks & Regards,
Erik

Re: Testsuite for avr-as

[Hans-Peter Nilsson]

On Fri, 10 Dec 2004, Erik Christiansen wrote:
> > > a) Figure out how to specify the assembler path, so we can regression
> > >    test a newly built one. (Doesn't seem to be an option in the *.d
> > >    head)
> >
> > No, you don't have to do this, the framework does it for you.
> > What I said should be all that's needed.  Did you change all
> > mmix-* to avr-*?
>
> Nope, I followed the Dec 07 advice, quoted at the top of this email,
> copying only mmix.exp to gas/testsuite/gas/avr/avr.exp, and performing
> the substitution. Was that the right thing to do?

Yes.

> Would you be able to hint how "the framework does it for you"?

All just-built gas are called $objdir/gas/as-new, and since
"make check" is run from $objdir/gas/testsuite, dejagnu doesn't
have to look very far.  Dejagnu knows about the name as-new.

Oh, did you mean checking an *installed* avr-gas, not the one in
objdir/gas?  Ok, that's a different thing.  I'm sure it can be
done, but I haven't looked up the details.  (Assuming you
configured for avr-unknown-elf, it should suffice, but perhaps
it doesn't, to have avr-unknown-elf-gas in your $PATH, and not
having as-new there.  I'm sure patches are welcome. ;-)

> Looking in the Makefile in obj-avr/gas, to find out where config is
> picked up, the "check" target becomes "check-recursive", which doesn't
> seem to be there. So now I'm digging through run_dump_test, since it
> must be revealed that way. (Eventually :-)

That detail isn't very revealing, methinks.  You missed the turn
at the $(RECURSIVE_TARGETS) rule which eventually lands on the
check-DEJAGNU rule, executing "runtest" with parameters.

The executable run from run_dump_test is just through the tcl
variable $AS, setup by the dejagnu framework (see proc find_gas
in dejagnu libgloss.exp -- yes, that file is a bit misnamed).

> > Usually people build in a directory other than where the source
> > is.  Maybe you don't do that and there are issues with running
> > the testsuite in the source directory.
>
> No way! Though I had been getting away with that for several years,
> there has been enough traffic on this topic, on several lists,
> indicating that it is not supported.

It should still work to build, at least for binutils, but
perhaps it's broken for "check".

All I said above I just looked up in the installed dejagnu and
objdir/gas.  There's no magic documentation path.  You would
eventually have found it out.  And I think that way would be
better too.

brgds, H-P

Re: Testsuite for avr-as

[Erik Christiansen]

Initial DejaGnu results for the avr target, running the generic gas
tests plus the first avr-specific .d file are:

      === gas Summary ===

# of expected passes    17
# of expected failures     1
# of unresolved testcases  2
../as-new 20041118                     # Thank you, Hans-Peter. :-)

After eliminating errors due to incomplete "regexification" of the new
opcode.d file, gas.log still reports "UNRESOLVED: opcode". No problem,
adding "set verbose 1" elucidates to this extent:

Error/Warning threshold exceeded:  1 0 (max. 1 3)

And "set verbose 3" additionally lists the regex/line pairs tested.
_But_, at no point is the error shown in the log. Is there a known way
to induce DejaGnu to reveal the "Error/Warning" that it claims to have
detected?

Erik

On Fri, Dec 10, 2004 at 07:15:08AM -0500, Hans-Peter Nilsson wrote:
> On Fri, 10 Dec 2004, Erik Christiansen wrote:
> > > > a) Figure out how to specify the assembler path, so we can regression
> > > >    test a newly built one. (Doesn't seem to be an option in the *.d
> > > >    head)
> > >
> > > No, you don't have to do this, the framework does it for you.
> > > What I said should be all that's needed.  Did you change all
> > > mmix-* to avr-*?
> >
> > Nope, I followed the Dec 07 advice, quoted at the top of this email,
> > copying only mmix.exp to gas/testsuite/gas/avr/avr.exp, and performing
> > the substitution. Was that the right thing to do?
> 
> Yes.
> 
> > Would you be able to hint how "the framework does it for you"?
> 
> All just-built gas are called $objdir/gas/as-new, and since
> "make check" is run from $objdir/gas/testsuite, dejagnu doesn't
> have to look very far.  Dejagnu knows about the name as-new.
> 
> Oh, did you mean checking an *installed* avr-gas, not the one in
> objdir/gas?  Ok, that's a different thing.  I'm sure it can be
> done, but I haven't looked up the details.  (Assuming you
> configured for avr-unknown-elf, it should suffice, but perhaps
> it doesn't, to have avr-unknown-elf-gas in your $PATH, and not
> having as-new there.  I'm sure patches are welcome. ;-)
> 
> > Looking in the Makefile in obj-avr/gas, to find out where config is
> > picked up, the "check" target becomes "check-recursive", which doesn't
> > seem to be there. So now I'm digging through run_dump_test, since it
> > must be revealed that way. (Eventually :-)
> 
> That detail isn't very revealing, methinks.  You missed the turn
> at the $(RECURSIVE_TARGETS) rule which eventually lands on the
> check-DEJAGNU rule, executing "runtest" with parameters.
> 
> The executable run from run_dump_test is just through the tcl
> variable $AS, setup by the dejagnu framework (see proc find_gas
> in dejagnu libgloss.exp -- yes, that file is a bit misnamed).
> 
> > > Usually people build in a directory other than where the source
> > > is.  Maybe you don't do that and there are issues with running
> > > the testsuite in the source directory.
> >
> > No way! Though I had been getting away with that for several years,
> > there has been enough traffic on this topic, on several lists,
> > indicating that it is not supported.
> 
> It should still work to build, at least for binutils, but
> perhaps it's broken for "check".
> 
> All I said above I just looked up in the installed dejagnu and
> objdir/gas.  There's no magic documentation path.  You would
> eventually have found it out.  And I think that way would be
> better too.
> 
> brgds, H-P

-- 
 _,-_|\    Erik Christiansen                    Phone: +61 3 9264 3416
/      \   Research & Development Division        Fax: +61 3 9264 3438
\_,-.__/   Voice Products Department
      v    NEC Business Solutions Pty. Ltd.     www.necbs.com.au

Re: Testsuite for avr-as

[Hans-Peter Nilsson]

On Thu, 20 Jan 2005, Erik Christiansen wrote:

> After eliminating errors due to incomplete "regexification" of the new
> opcode.d file, gas.log still reports "UNRESOLVED: opcode". No problem,
> adding "set verbose 1" elucidates to this extent:

Usually you add one "-v" for each verbosity level to
RUNTESTFLAGS on the command line to the same effect.

> Error/Warning threshold exceeded:  1 0 (max. 1 3)
>
> And "set verbose 3" additionally lists the regex/line pairs tested.
> _But_, at no point is the error shown in the log. Is there a known way
> to induce DejaGnu to reveal the "Error/Warning" that it claims to have
> detected?

I don't know, I don't recognize the error you mention.  If you
use run_dump_test, the lines that differ and the expected
pattern are shown in the log.  But it seems this is not what you
see, but some other error?

I suggest you send your testsuite patches to the list.
Maybe it's a bug in one of the regexps in the test.

brgds, H-P

Re: Testsuite for avr-as

[Erik Christiansen]

On Thu, Jan 20, 2005 at 07:59:42AM -0500, Hans-Peter Nilsson wrote:
> I suggest you send your testsuite patches to the list.
> Maybe it's a bug in one of the regexps in the test.

Thanks. If I can't crack it in the next few days, I'll do that.

Erik