The use of the name tmpdir in the tests

The use of the name tmpdir in the tests

[Ruud van der Pas]

Hi team,

This is an absolute detail, but as I am working on the test suite
for gprofng, I noticed the use of a directory called "tmpdir".

I count about 155 .exp files across binutils that use it:

$ grep -sw tmpdir `find . -name "*".exp -print` | awk 'BEGIN {FS=":"} {print $1}' | uniq | wc
    155     155    6000
$

The thing is that in the case of gprofng at least, this is far from
a temporary directory. It is going to contain the generated experiment
data that is crucial for troubleshooting in case of a failure.

Of course we could place those experiments elsewhere, but my idea
was to give tmpdir a more meaningful name.

Are there any objections against doing so? For example, will 
certain scripts fail?

If not, any suggestion what would be a good name? I'm not that
creative so don't get any further than "results" or "expresults"
and am definitely open to suggestions.

Kind regards, Ruud

Re: The use of the name tmpdir in the tests

[Jan Beulich]

On 15.06.2023 23:53, Ruud van der Pas via Binutils wrote:
> Hi team,
> 
> This is an absolute detail, but as I am working on the test suite
> for gprofng, I noticed the use of a directory called "tmpdir".
> 
> I count about 155 .exp files across binutils that use it:
> 
> $ grep -sw tmpdir `find . -name "*".exp -print` | awk 'BEGIN {FS=":"} {print $1}' | uniq | wc
>     155     155    6000
> $
> 
> The thing is that in the case of gprofng at least, this is far from
> a temporary directory. It is going to contain the generated experiment
> data that is crucial for troubleshooting in case of a failure.
> 
> Of course we could place those experiments elsewhere, but my idea
> was to give tmpdir a more meaningful name.
> 
> Are there any objections against doing so? For example, will 
> certain scripts fail?
> 
> If not, any suggestion what would be a good name? I'm not that
> creative so don't get any further than "results" or "expresults"
> and am definitely open to suggestions.

But that would be misleading too, if applied tree-wide, since often
it's indeed just temporary data that is put there. Iirc some tests
leave stuff there (intentionally or not), while others remove things
after completion. So you keeping things in there would be within
what's done elsewhere. Plus still the test results are temporary
data; it's needed in case of problems, but not typically otherwise.

Jan

Re: The use of the name tmpdir in the tests

[Ruud van der Pas]

Hi Jan,

Thanks for the response.

> But that would be misleading too, if applied tree-wide, since often
> it's indeed just temporary data that is put there. Iirc some tests
> leave stuff there (intentionally or not), while others remove things
> after completion. So you keeping things in there would be within
> what's done elsewhere. Plus still the test results are temporary
> data; it's needed in case of problems, but not typically otherwise.

That is true.

Kind regards, Ruud

Re: The use of the name tmpdir in the tests

[Nick Clifton]

Hi Ruud,

> This is an absolute detail, but as I am working on the test suite
> for gprofng, I noticed the use of a directory called "tmpdir".

> The thing is that in the case of gprofng at least, this is far from
> a temporary directory. It is going to contain the generated experiment
> data that is crucial for troubleshooting in case of a failure.

Wouldn't it be better for gprofng to store the data in a directory
named after the experiment ?  That way if the user runs multiple,
different experiments it would still be possible to determine which
directory contains the relevant data.



> Of course we could place those experiments elsewhere, but my idea
> was to give tmpdir a more meaningful name.
> 
> Are there any objections against doing so? For example, will
> certain scripts fail?

Probably.  There are lots of places in the binutils sources where
the "tmpdir" directory name is explicitly referenced, so you would
need a large patch to change them all.  Plus I would not be at all
surprised to find that there are users of the binutils who expect
to find certain things in the tmpdir directories.  So renaming
would definitely cause them problems.


> If not, any suggestion what would be a good name? I'm not that
> creative so don't get any further than "results" or "expresults"
> and am definitely open to suggestions.

How about "tmpdir-<experiment_name>" or "resultdir-<experiment_name>" ?

Cheers
   Nick

Re: The use of the name tmpdir in the tests

[Ruud van der Pas]

Hi Nick,

Thanks very much for the response!

> Wouldn't it be better for gprofng to store the data in a directory
> named after the experiment ?  That way if the user runs multiple,
> different experiments it would still be possible to determine which
> directory contains the relevant data.

Thanks for the suggestion!

We have tried different approaches and given the kind of things
we would like to test (e.g. comparison of experiments), we came
up with a structure where each code/test has its own main directory
in tmpdir. It is named after the test (e.g. "mxv").

Within this directory there will be one or more subdirectories
with a name that is set in the .exp file for that test.

For example, in my current set up, this is what you see in tmpdir:

$ ls tmpdir
mxv/
$ ls tmpdir/mxv
no-opt/  opt/
$

This means there are two different main tests for the "mxv" code
and as suggested by the name, one is for an executable compiled
without optimizations enabled and the other one uses an executable
compiled with certain optimization flags.

The compiler flags are set in the same .exp file by the way.

This is an example line from the table:

{ "mxv"  "opt"  "-g -O"  "-p on"  "-m 4000 -n 2000" "2" "4"}

The last two numbers are the initial and end value for the
thread count used. This example will run the experiments
for 2, 3, and 4 threads. One thing I'm considering is to also
include an increment to reduce on the number of cases.

Within each subdirectory, there are one or more log and result files,
plus the corresponding experiment directories. It also contains the
generated gprofng script file that has been used to generate the results.

The log files are named after the functionality (e.g. collect-app.log).
The result files include the name of the gprofng tool and command
used, plus the number of threads (e.g. display-text.functions.2.thr.txt).

The name of the experiment directory uses some of the settings
in the table that defines the experiment. In particular, the
name of the test and the number of threads used. For example
mxv.2.thr.er.

Since we are in a subdirectory we already know more, like "no-opt",
and by keeping the name short we avoid cluttering the directory
with (too) long names.

We typically want to extract more than one result from an experiment
directory, because we'd like to test various commands (e.g. -header,
-threads, -functions, etc.) and eventually also different tools.

Essentially, these subdirectories contain everything needed to 
manually reproduce the results (the generated script file handles
everything) and/or conduct some different "what if" experiments.

In this way, we hope to make it easier to quickly find the test(s)
that failed (e.g. mxv/no-opt) and ease the troubleshooting.

At this point, I have this framework working, but have not yet
gotten to work on the verification part. Which is of course the
crucial next step.

> Probably.  There are lots of places in the binutils sources where
> the "tmpdir" directory name is explicitly referenced, so you would
> need a large patch to change them all.  Plus I would not be at all
> surprised to find that there are users of the binutils who expect
> to find certain things in the tmpdir directories.  So renaming
> would definitely cause them problems.

Thanks. It is clear then :-) We should not change that name.

> How about "tmpdir-<experiment_name>" or "resultdir-<experiment_name>" ?

Thanks for the suggestions!

With the above structure, I hope we make it easy for people to do the
troubleshooting, but as always, suggestions are very welcome!

Kind regards, Ruud