[Bug testsuite/39952] New: Inadequate gcc.dg/compat/struct-layout-1_generate.c

[Bug testsuite/39952] New: Inadequate gcc.dg/compat/struct-layout-1_generate.c

[hjl dot tools at gmail dot com]

gcc.dg/compat/struct-layout-1_generate.c generates some tests
randomly. Not all possible combinations are tested. The test

T(2456,double a;struct{int * c;}b[];,F(2456,a,-85022.609375,131929.593750))

from PR 39903 was never generated on Linux/x86-64. As the result,
PR 39903 regression wasn't detected on Linux/x86-64.


-- 
           Summary: Inadequate gcc.dg/compat/struct-layout-1_generate.c
           Product: gcc
           Version: 4.5.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: testsuite
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: hjl dot tools at gmail dot com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952

[Bug testsuite/39952] Inadequate gcc.dg/compat/struct-layout-1_generate.c

[jakub at gcc dot gnu dot org]

------- Comment #1 from jakub at gcc dot gnu dot org  2009-04-29 14:04 -------
That's by design.
Obviously there are so many possible combinations that you can't exhaustively
test them all, that's why this test randomly chooses some.
You can pass -n count to the generator to generate more (or fewer) tests,
instead of the default 3000.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952

[Bug testsuite/39952] Inadequate gcc.dg/compat/struct-layout-1_generate.c

[hjl dot tools at gmail dot com]

------- Comment #2 from hjl dot tools at gmail dot com  2009-04-29 14:40 -------
(In reply to comment #1)
> That's by design.
> Obviously there are so many possible combinations that you can't exhaustively
> test them all, that's why this test randomly chooses some.
> You can pass -n count to the generator to generate more (or fewer) tests,
> instead of the default 3000.
> 

If it is truly random, shouldn't someone on Linux/x86-64 run into it
sometimes?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952

[Bug testsuite/39952] Inadequate gcc.dg/compat/struct-layout-1_generate.c

[pinskia at gcc dot gnu dot org]

------- Comment #3 from pinskia at gcc dot gnu dot org  2009-04-29 14:44 -------
(In reply to comment #2)
> 
> If it is truly random, shouldn't someone on Linux/x86-64 run into it
> sometimes?

it uses the same seed each time.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952

[Bug testsuite/39952] Inadequate gcc.dg/compat/struct-layout-1_generate.c

[hjl dot tools at gmail dot com]

------- Comment #4 from hjl dot tools at gmail dot com  2009-04-29 14:56 -------
(In reply to comment #3)
> (In reply to comment #2)
> > 
> > If it is truly random, shouldn't someone on Linux/x86-64 run into it
> > sometimes?
> 
> it uses the same seed each time.
> 

Why can't we use the current time or PID as seed?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952

[Bug testsuite/39952] Inadequate gcc.dg/compat/struct-layout-1_generate.c

[jakub at gcc dot gnu dot org]

------- Comment #5 from jakub at gcc dot gnu dot org  2009-04-29 14:59 -------
Yeah, and that's because otherwise reproduceability would be a nightmare.
We even use the same pseudo-random generator on all hosts to have as few
differences as possible.
It is certainly possibility to add a -sSEED option (and -s meaning
generate_srandom call seeing with time_t or something), but it isn't something
that should be run by default.

If you have sufficient CPU cycles, you can obviously in your testing bump -n
from 3000 to 1000000 or something and use -s, after making sure you make a copy
of the generated files whenever you notice some failure to be able to reproduce
it.  Generally, a single line like in struct-layout-1_test.h on which the test
failed should be enough to reproduce it.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952

[Bug testsuite/39952] Inadequate gcc.dg/compat/struct-layout-1_generate.c

[hjl dot tools at gmail dot com]

------- Comment #6 from hjl dot tools at gmail dot com  2009-04-29 15:48 -------
So the issues are

1. We want to test as many different combination as possible at random.
2. We want each test reproducible.

Can we generate tests at random in such a way that they are reproducible?
Can we do

1. Put the random seed in generated file.
2. Change struct-layout-1_generate.c to take a random seed.

so that if there is a failure, we can run struct-layout-1_generate.c
with the known random seed on the same platform?


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952

[Bug testsuite/39952] Inadequate gcc.dg/compat/struct-layout-1_generate.c

[joseph at codesourcery dot com]

------- Comment #7 from joseph at codesourcery dot com  2009-04-29 15:56 -------
Subject: Re:  Inadequate gcc.dg/compat/struct-layout-1_generate.c

On Wed, 29 Apr 2009, hjl dot tools at gmail dot com wrote:

> So the issues are
> 
> 1. We want to test as many different combination as possible at random.
> 2. We want each test reproducible.
> 
> Can we generate tests at random in such a way that they are reproducible?
> Can we do
> 
> 1. Put the random seed in generated file.
> 2. Change struct-layout-1_generate.c to take a random seed.
> 
> so that if there is a failure, we can run struct-layout-1_generate.c
> with the known random seed on the same platform?

If you wish to contribute extra testing of these tests you can certainly 
do such things.  The default should be results that are comparable between 
different runs of the same test (the thing after "PASS: " or "FAIL: ") at 
different times by the same person or by different people for the same 
target, so that when results showing a failure appear on gcc-testresults 
it is unambiguous from the FAIL lines there exactly what was failing and 
the next run for that target will show the same lines if the bug is not 
fixed.

If you wish to do such extra testing it would be useful for you to file 
bug reports with reduced testcases showing the failures.  Such reports are 
useful however the tests were generated (including tests generated by 
tools not included in GCC, if the licenses on those tools permit the 
distribution of the tests); certainly some bugs people report appear to 
have tests generated by unspecified tools they may have developed.


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39952