gprofng testing

gprofng testing

[Vladimir Mezentsev]

Hi All,

We have many tests for gprofng inside Oracle.
Only 3 tests are public and they are in binutils-gdb.git/gprofng/testsuite.

I want to add more tests for gprofng.
These tests use pre-existing experiments and binaries.
Each test has a golden file which we compare against the new gprofng output.

We never rebuilt the experiments and binaries. But sometimes we changed 
the golden file.
The size of experiments can be big.
The size of gold files is usually small.

How to make such tests public?

Thank you,
-Vladimir

PS.
I see that the old gprof doesn't have tests inside binutils-gdb.git. How 
do we test gprof?

Re: gprofng testing

[Nick Clifton]

Hi Vladimir,

> I want to add more tests for gprofng.

That would be most appreciated.

> These tests use pre-existing experiments and binaries.
> Each test has a golden file which we compare against the new gprofng output.
> 
> We never rebuilt the experiments and binaries. But sometimes we changed the golden file.
> The size of experiments can be big.
> The size of gold files is usually small.
> 
> How to make such tests public?

With difficulty.  The size of the experiment files will be an obstacle
I expect.  Is it possible to instead provide a source file and have the
test compile it first instead ?

Assuming that the answer to that question is no, then is it possible
to create smaller test cases, or to compress the test files files and
then decompress them when they are being used ?

My best suggestion would be to create lots of small tests rather than
several large tests.  If that is possible.

> PS.
> I see that the old gprof doesn't have tests inside binutils-gdb.git. How do we test gprof?

We don't. :-(

Gprof is very old and was added to the binutils long ago.  Long before
testsuites were invented.  OK, so that is not quite true, testsuites
have been around for as along as code has been around, but the gprof
sources were added to the binutils without a testsuite and noone has
seen fit to create one since.  Of course if you are volunteering, I
will not say "no"...

Cheers
   Nick

Re: gprofng testing

[Vladimir Mezentsev]

Hi Nick,
thank you for the answer.
My comments are below.

On 5/10/23 04:20, Nick Clifton wrote:
> Hi Vladimir,
>
>> I want to add more tests for gprofng.
>
> That would be most appreciated.
>
>> These tests use pre-existing experiments and binaries.
>> Each test has a golden file which we compare against the new gprofng 
>> output.
>>
>> We never rebuilt the experiments and binaries. But sometimes we 
>> changed the golden file.
>> The size of experiments can be big.
>> The size of gold files is usually small.
>>
>> How to make such tests public?
>
> With difficulty.  The size of the experiment files will be an obstacle
> I expect.  Is it possible to instead provide a source file and have the
> test compile it first instead ?

This is exactly what we have now in binutils-gdb.git.
We compare the time for each function in the application output and the 
gprofng report.
The small discrepancy is OK.

But we cannot check the correctness of the gprofng report, such as the 
correct alignment of metrics or the order of functions.


>
> Assuming that the answer to that question is no, then is it possible
> to create smaller test cases, or to compress the test files files and
> then decompress them when they are being used ?
  Is 2-4 megabyte small ?
In any case, it was only on large experiments that we saw problems with 
the synchronization and reallocation of our tables.

>
> My best suggestion would be to create lots of small tests rather than
> several large tests.  If that is possible.
>
>> PS.
>> I see that the old gprof doesn't have tests inside binutils-gdb.git. 
>> How do we test gprof?
>
> We don't. :-(

I see that gdb/testsuite is small.
Is there a big public test suite for gdb (or any other binutils/gcc 
components) ?

Thank you,
-Vladimir


> Gprof is very old and was added to the binutils long ago.  Long before
> testsuites were invented.  OK, so that is not quite true, testsuites
> have been around for as along as code has been around, but the gprof
> sources were added to the binutils without a testsuite and noone has
> seen fit to create one since.  Of course if you are volunteering, I
> will not say "no"...
>
> Cheers
>   Nick
>
>

Re: gprofng testing

[Nick Clifton]

Hi Vladimir,

>>> The size of experiments can be big.

>> With difficulty.  The size of the experiment files will be an obstacle
>> I expect.  Is it possible to instead provide a source file and have the
>> test compile it first instead ?
> 
> This is exactly what we have now in binutils-gdb.git.
> We compare the time for each function in the application output and the gprofng report.
> The small discrepancy is OK.
> 
> But we cannot check the correctness of the gprofng report, such as the correct alignment of metrics or the order of functions.

Why not ?  How about instead of compiling examples, you start with
assembler sources (possibly based upon compiled source code).  This
would remove any dependencies upon compiler behaviour, code layout
and so on.


>> Assuming that the answer to that question is no, then is it possible
>> to create smaller test cases, or to compress the test files files and
>> then decompress them when they are being used ?
>   Is 2-4 megabyte small ?

No. :-(

> In any case, it was only on large experiments that we saw problems with the synchronization and reallocation of our tables.

Well at least you caught these problems yourselves.


> Is there a big public test suite for gdb (or any other binutils/gcc components) ?

No.

Well unless you consider the various Linux and BSD distributions to testsuites.
After all if you put a broken linker into the build root of a distribution you
quickly get a lot of people complaining at you.  (I speak from experience...)

But basically the answer is that there is currently nowhere that hosts large
tests of any of the GNU tools.  It would be nice to have such a resource, of
course, but I do not see one appearing soon.

Cheers
   Nick