RFC - changes to the test suite

RFC - changes to the test suite

[Tom Tromey]

I've been working for a while on file-level parallelism for the test
suite.  The idea is to make each .exp file independent, so that we can
run the test cases in parallel at whatever granularity we like.  This
will let the test suite scale better on more powerful machines.

Most of the needed changes to the tests themselves are straightforward:
adding standard_testfile and standard_output_file in various places, so
that tests no longer stomp on each others' executables, and so that the
location of output files can be controlled.

However, this latter point gets to something I thought I would bring up
for discussion before prepping this (somewhat large[*]) series for
submission.

In the interest of nicer test isolation, I changed where the test suite
places most files.  For a given test file, say gdb.DIR/FILE.exp, the
various output files (executables, .o files, libraries, etc) go in a new
directory named "./outputs/gdb.DIR/FILE".

This has two nice effects.

First, we don't have to audit every single intermediate file to look for
basename clashes -- we can just make sure to use standard_output_file.

Second, it means all the "clean" rules can be replaced with "rm -rf
outputs".  This resulted in a nice cleanup -- I got rid of all the (IMO
barely maintained) Makefile.in files underneath gdb/testsuite.  This in
turn speeds up configure a little.


So, the first question is -- does anybody care strongly about where the
files end up?  And, if you do care, why do you care?

I can pretty easily make it so that the files end up in the usual place
when not running in "parallel" mode.  I don't think this is as good, but
I could do it if there is an outcry.


There are a few other changes, too, but nothing quite as visible.

I changed various gdb-property-testing procs to optionally cache their
results in a "cache" directory.  That way we aren't re-running things
like the compile check in support_complex_tests for every .exp file that
might need it.

I also added an "inotify" mode to the tests so you can easily see which
tests write files outside of their specified directory.  There are still
a few remaining, I ran out of steam dotting every "i".

Comments?

I plan to start organizing this branch for submission soon.

Tom

[*]  329 files changed, 2099 insertions(+), 4847 deletions(-)

Re: RFC - changes to the test suite

[Joel Brobecker]

> In the interest of nicer test isolation, I changed where the test suite
> places most files.  For a given test file, say gdb.DIR/FILE.exp, the
> various output files (executables, .o files, libraries, etc) go in a new
> directory named "./outputs/gdb.DIR/FILE".
> 
> This has two nice effects.

IMO, that would be a great improvement. This renders moot the exact
problem you are mentioning of trying to cleanup the gdb.ada directory.
In terms of the directory name itself, no preference.

-- 
Joel

Re: RFC - changes to the test suite

[Jan Kratochvil]

On Mon, 18 Feb 2013 18:57:19 +0100, Tom Tromey wrote:
> So, the first question is -- does anybody care strongly about where the
> files end up?  And, if you do care, why do you care?
> 
> I can pretty easily make it so that the files end up in the usual place
> when not running in "parallel" mode.  I don't think this is as good, but
> I could do it if there is an outcry.

If it is easy enough I find it useful.

The reasons are the same why I like in-source-tree-builds.  I have everything
on the same place, no need to <tab>-complete source directory and
<tab>-complete different output directory and still dealing with two
directories while editing sources and checking their output.

This is useful only when running single file by "runtest x/y.exp".
In some batch parallel mode I sure do not mind where the output files end up.


Thanks,
Jan

Re: RFC - changes to the test suite

[Tom Tromey]

>>>>> "Jan" == Jan Kratochvil <jan.kratochvil@redhat.com> writes:

Jan> This is useful only when running single file by "runtest x/y.exp".
Jan> In some batch parallel mode I sure do not mind where the output
Jan> files end up.

I wonder then whether you care about "make clean" working in this
situation.  My current patches basically gut it.

Tom

Re: RFC - changes to the test suite

[Jan Kratochvil]

On Mon, 18 Feb 2013 21:46:42 +0100, Tom Tromey wrote:
> >>>>> "Jan" == Jan Kratochvil <jan.kratochvil@redhat.com> writes:
> 
> Jan> This is useful only when running single file by "runtest x/y.exp".
> Jan> In some batch parallel mode I sure do not mind where the output
> Jan> files end up.
> 
> I wonder then whether you care about "make clean" working in this
> situation.

No.


Thanks,
Jan

Re: RFC - changes to the test suite

[Yao Qi]

Tom,
overall, I like it.  Two questions below,

On 02/19/2013 01:57 AM, Tom Tromey wrote:
> So, the first question is -- does anybody care strongly about where the
> files end up?  And, if you do care, why do you care?

When running testsuite with boardfile native-gdbserver.exp, does 
GDBserver need to know the updated location of files? or do we plan to 
spawn GDBserver in "./outputs/gdb.DIR/FILE" directory?

>
> I also added an "inotify" mode to the tests so you can easily see which
> tests write files outside of their specified directory.  There are still
> a few remaining, I ran out of steam dotting every "i".

What is the motivation of adding "inotify" mode? to monitor the progress 
of a given test, or something else?

-- 
Yao (齐尧)

Re: RFC - changes to the test suite

[Tom Tromey]

Tom> So, the first question is -- does anybody care strongly about where the
Tom> files end up?  And, if you do care, why do you care?

Yao> When running testsuite with boardfile native-gdbserver.exp, does
Yao> GDBserver need to know the updated location of files? or do we plan to
Yao> spawn GDBserver in "./outputs/gdb.DIR/FILE" directory?

I will test it and find out.

Tom> I also added an "inotify" mode to the tests so you can easily see which
Tom> tests write files outside of their specified directory.  There are still
Tom> a few remaining, I ran out of steam dotting every "i".

Yao> What is the motivation of adding "inotify" mode? to monitor the
Yao> progress of a given test, or something else?

It is just for debugging the code to ensure that tests don't write
outside of the allowed directories.  It isn't perfect, but it helped me
find some bugs, and I thought it would be handy to keep around.

Tom