Meta-expectations when testing

Meta-expectations when testing

[Hugh Sasse Staff Elec Eng]

As far as I can gather from the listing on the GCC tests page
GCC-3.2.1 should build OK on Solaris2.9, and pass the tests.
However on my system I get the following results at the end of
running make check, and I don't know if the deviations are
significant or not.
        [...[
bstdc++-v3 Summary ===

# of expected passes            410
# of unexpected failures        1
# of unexpected successes       10
# of expected failures          16
gmake[4]: *** [check-DEJAGNU] Error 1
gmake[4]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3/testsuite'
gmake[3]: *** [check-am] Error 2
gmake[3]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3/testsuite'
gmake[2]: *** [check-recursive] Error 1
gmake[2]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3'
gmake[1]: *** [check-target-libstdc++-v3] Error 2
gmake[1]: Leaving directory `/scratch/hgs/gcc-build'
gmake: *** [check] Error 2
neelix hgs 728 %> setenv
        [...]
CC=gcc
DEJAGNULIBS=/usr/local/share/dejagnu
TCL_LIBRARY=/usr/local/lib/tcl8.3
neelix hgs 729 %>

Built with ../gcc-3.2.1/configure --with-as=/usr/local/bin/as
--with-ld=/usr/local/bin/ld and using the latest binutils
(../binutils-2.13.2.1) and building with the Sun supplied
GCC 2.95.3
On an Enterprise 250 (sparc v9) Solaris 2.9 system.

Should I "expect the unexpected" failures or not?  There is some
evidence in the documentation on the testing pages that this testing
code is not as up-to-date as they maintainers would wish, (given the
usual constraints on volunteer based softare development), so this
may not be a daft a question as it looks :-) I hope!

        Thank you,
        Hugh

Re: Meta-expectations when testing

[Janis Johnson]

On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> 
> As far as I can gather from the listing on the GCC tests page
> GCC-3.2.1 should build OK on Solaris2.9, and pass the tests.
> However on my system I get the following results at the end of
> running make check, and I don't know if the deviations are
> significant or not.
>         [...[
> bstdc++-v3 Summary ===
> 
> # of expected passes            410
> # of unexpected failures        1
> # of unexpected successes       10
> # of expected failures          16

Almost every target has some unexpected failures.

> gmake[4]: *** [check-DEJAGNU] Error 1
> gmake[4]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3/testsuite'
> gmake[3]: *** [check-am] Error 2
> gmake[3]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3/testsuite'
> gmake[2]: *** [check-recursive] Error 1
> gmake[2]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3'
> gmake[1]: *** [check-target-libstdc++-v3] Error 2
> gmake[1]: Leaving directory `/scratch/hgs/gcc-build'
> gmake: *** [check] Error 2

This is normal when there are some tests that fail.  Use "make -k check"
so that make will ignore errors.

> Should I "expect the unexpected" failures or not?  There is some
> evidence in the documentation on the testing pages that this testing
> code is not as up-to-date as they maintainers would wish, (given the
> usual constraints on volunteer based softare development), so this
> may not be a daft a question as it looks :-) I hope!

The GCC build status lists have links to archived test results for many
targets, so you can often find a similar target with test results to
compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.

Janis

Re: Meta-expectations when testing

[Hugh Sasse Staff Elec Eng]

On Mon, 24 Feb 2003, Janis Johnson wrote:

> On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> >
> > As far as I can gather from the listing on the GCC tests page
> > GCC-3.2.1 should build OK on Solaris2.9, and pass the tests.
> > However on my system I get the following results at the end of
        [...]
> > # of expected passes            410
> > # of unexpected failures        1
> > # of unexpected successes       10
> > # of expected failures          16
>
> Almost every target has some unexpected failures.

OK, than how is one to tell which are significant?
>
> > gmake[4]: *** [check-DEJAGNU] Error 1
        [...]
> > gmake: *** [check] Error 2
>
> This is normal when there are some tests that fail.  Use "make -k check"
> so that make will ignore errors.

OK, I forgot the -k, but I suspect that some errors should not be
ignored.
>
> > Should I "expect the unexpected" failures or not?  There is some
> > evidence in the documentation on the testing pages that this testing
> > code is not as up-to-date as they maintainers would wish, (given the
> > usual constraints on volunteer based softare development), so this
> > may not be a daft a question as it looks :-) I hope!
>
> The GCC build status lists have links to archived test results for many
> targets, so you can often find a similar target with test results to
> compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.

I found the same target and it said the build was successful. Other
similar cases (3.2.2) referred to errors.  Therefore I concluded
that I should get no errors.  Hence my question as to how to
determine if the discrepancy is significant.  I can see that this is
not a simple question because the significance may well depend on
what software one is building with GCC.
>
> Janis
>
        Thank you,
        Hugh

Re: Meta-expectations when testing

[Janis Johnson]

On Tue, Feb 25, 2003 at 10:41:57AM +0000, Hugh Sasse Staff Elec Eng wrote:
> On Mon, 24 Feb 2003, Janis Johnson wrote:
> > On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> > > Should I "expect the unexpected" failures or not?  There is some
> > > evidence in the documentation on the testing pages that this testing
> > > code is not as up-to-date as they maintainers would wish, (given the
> > > usual constraints on volunteer based softare development), so this
> > > may not be a daft a question as it looks :-) I hope!
> >
> > The GCC build status lists have links to archived test results for many
> > targets, so you can often find a similar target with test results to
> > compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.
> 
> I found the same target and it said the build was successful. Other
> similar cases (3.2.2) referred to errors.  Therefore I concluded
> that I should get no errors.  Hence my question as to how to
> determine if the discrepancy is significant.  I can see that this is
> not a simple question because the significance may well depend on
> what software one is building with GCC.

Follow the links for test results.  Many people report successful builds
without even running the test suites, let alone submitting the test
results, and having some unexpected test failures wouldn't cause someone
to report that the build had problems.  If there are no test results for
your GCC version and target, look at results for the same version but
slightly different targets.

Ideally, in released versions of GCC all tests that are known to fail
for a particular target would be marked as expected failures, but that
requires a lot of work and seldom happens.

Janis

Re: Meta-expectations when testing

[Hugh Sasse Staff Elec Eng]

On Tue, 25 Feb 2003, Janis Johnson wrote:

> On Tue, Feb 25, 2003 at 10:41:57AM +0000, Hugh Sasse Staff Elec Eng wrote:
> > On Mon, 24 Feb 2003, Janis Johnson wrote:
> > > On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> > > > Should I "expect the unexpected" failures or not?  There is some
        [...]
> > > The GCC build status lists have links to archived test results for many
> > > targets, so you can often find a similar target with test results to
> > > compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.
> >
> > I found the same target and it said the build was successful. Other
        [...]
>
> Follow the links for test results.  Many people report successful builds
> without even running the test suites, let alone submitting the test

This was the case here. Thanks for pointing that out.

I'd like to propose the following patch to the test.html:
Its purposes: to point that out and to put a reference in there to
the list of results, which doesn't seem to be on the test page.

--- test.html.orig      2003-02-25 18:49:09.360007000 +0000
+++ test.html   2003-02-25 18:55:30.920007000 +0000
@@ -178,6 +178,12 @@
 over whether or not a test is expected to fail.  We expect to fix this
 problem in future releases.

+   <p>You may find it useful to compare your results with those from
+similar builds listed at <a
+href="http://gcc.gnu.org/buildstat.html">http://gcc.gnu.org/buildstat.html</a>,
+but be aware that sometimes a build listed as successful has no test
+results associated with it.
+
 <h3 class="section"><a name="TOC3"></a>Submitting test results</h3>

    <p>If you want to report the results to the GCC project, use the



I just tried to submit my results and it failed to work:
I had no comments to make as such but to keep the command the same I
did
touch no_comment
../gcc-3.2.1/contrib/test_summary -p no_comment -m gcc-testresults@gcc.gnu.org |sh
tail: cannot open input

        [...]
>
> Ideally, in released versions of GCC all tests that are known to fail
> for a particular target would be marked as expected failures, but that
> requires a lot of work and seldom happens.

Yes, I understand that.
>
> Janis
>
        Thank you,
        Hugh

Re: Meta-expectations when testing

[Janis Johnson]

On Wed, Feb 26, 2003 at 09:59:40AM +0000, Hugh Sasse Staff Elec Eng wrote:
> On Tue, 25 Feb 2003, Janis Johnson wrote:
[snip]
> >
> > Follow the links for test results.  Many people report successful builds
> > without even running the test suites, let alone submitting the test
> 
> This was the case here. Thanks for pointing that out.
> 
> I'd like to propose the following patch to the test.html:
> Its purposes: to point that out and to put a reference in there to
> the list of results, which doesn't seem to be on the test page.
> 
> --- test.html.orig      2003-02-25 18:49:09.360007000 +0000
> +++ test.html   2003-02-25 18:55:30.920007000 +0000
> @@ -178,6 +178,12 @@
>  over whether or not a test is expected to fail.  We expect to fix this
>  problem in future releases.
> 
> +   <p>You may find it useful to compare your results with those from
> +similar builds listed at <a
> +href="http://gcc.gnu.org/buildstat.html">http://gcc.gnu.org/buildstat.html</a>,
> +but be aware that sometimes a build listed as successful has no test
> +results associated with it.
> +
>  <h3 class="section"><a name="TOC3"></a>Submitting test results</h3>
> 
>     <p>If you want to report the results to the GCC project, use the
 
The first sentence in test.html (generated from gcc/doc/install.texi) is:

  Before you install GCC, we encourage you to run the testsuites and to
  compare your results with results from a similar configuration that have
  been submitted to the @uref{http://gcc.gnu.org/ml/gcc-testresults/,,
  gcc-testresults mailing list}.

Would appending the following to that be sufficient?

  Some of these archived results are linked from the build status lists
  at @uref{http://gcc.gnu.org/buildstat.html}, although not everyone who
  reports a successful build runs the testsuites and submits the results. 

(Also, the maintainer of the build status lists sometimes doesn't have
time to go through the gcc-testresults archives after a release to
include new reports in the build status list as she ought to, and
welcomes mail pointing out the URLs of such reports, or patches to the
status list.)

To me, it seems clear enough already that a successful build doesn't
imply that all of the tests passed.

> 
> I just tried to submit my results and it failed to work:
> I had no comments to make as such but to keep the command the same I
> did
> touch no_comment
> ../gcc-3.2.1/contrib/test_summary -p no_comment -m gcc-testresults@gcc.gnu.org |sh
> tail: cannot open input
> 
>         [...]

The script moves the *.sum and *.log files to new names, so you can only
run it once unless you move the files back to their original names.  You
shouldn't need to supply any options, since the default is to mail to
the address you specified.  If you direct the output to a file then you
can look it over before running it, and that way it's available to
modify and run again if there were problems the first time.

Janis

Re: Meta-expectations when testing

[Hugh Sasse Staff Elec Eng]

On Wed, 26 Feb 2003, Janis Johnson wrote:

> On Wed, Feb 26, 2003 at 09:59:40AM +0000, Hugh Sasse Staff Elec Eng wrote:
> > On Tue, 25 Feb 2003, Janis Johnson wrote:
> [snip]
> > >
> > > Follow the links for test results.  Many people report successful builds
> > > without even running the test suites, let alone submitting the test
> >
> > This was the case here. Thanks for pointing that out.
> >
> > I'd like to propose the following patch to the test.html:
> > Its purposes: to point that out and to put a reference in there to
> > the list of results, which doesn't seem to be on the test page.
> >
        [patch elided.
>
> The first sentence in test.html (generated from gcc/doc/install.texi) is:
>
>   Before you install GCC, we encourage you to run the testsuites and to
>   compare your results with results from a similar configuration that have
>   been submitted to the @uref{http://gcc.gnu.org/ml/gcc-testresults/,,
>   gcc-testresults mailing list}.

Yes, but that is on the install page.  When you are testing you will
have the testing page open.  I think that one wants this information
when one runs the tests, and before one has exhauseted all the
possibilities of what has gone wrong in running the tests
[forgetting -k, not haveing the

         TCL_LIBRARY = /usr/local/share/tcl8.0
         DEJAGNULIBS = /usr/local/share/dejagnu

set, and so forth], because when one gets to the bit about submitting
one's test results one wants to submit something sensible.  I could
never remember where I'd seen it, and it was a job finding it again.
[Sometimes the linking in the web feels like "a lot of twisty little
passages, all alike" :-)]

I know one should not repeat information unnecessarily (because of
correcting it later) and HTML has no include facility, but I found
having this information on another page rather inconvenient.

I would be satisfied if, to avoid duplication, there was a link "up"
to install.html#where_the_buildstats_are, or something like that.


> Would appending the following to that be sufficient?
>
>   Some of these archived results are linked from the build status lists
>   at @uref{http://gcc.gnu.org/buildstat.html}, although not everyone who
>   reports a successful build runs the testsuites and submits the results.

Not on the install page, for the reasons above. I really believe
that the stepwise approach on the install page, (download,
configure, build, test, install) should obviate the need to go
backwards to find information.

I don't build GCC very often, and don't consider myself familiar
with the details of the process, and anything that makes it easier
for people in my position will reduce traffic to the list.
>
> (Also, the maintainer of the build status lists sometimes doesn't have
> time to go through the gcc-testresults archives after a release to
> include new reports in the build status list as she ought to, and
> welcomes mail pointing out the URLs of such reports, or patches to the
> status list.)

There seems to be no address for a person on these pages, only for
groups of people {gcc,gnu}@gnu.org, so I don't know who that is.
>
> To me, it seems clear enough already that a successful build doesn't
> imply that all of the tests passed.

Yes, but the question I posed was about how to interpret the
failures.  The "similar platforms" is an effective answer to this.
>
> >
> > I just tried to submit my results and it failed to work:
> > I had no comments to make as such but to keep the command the same I
> > did
> > touch no_comment
> > ../gcc-3.2.1/contrib/test_summary -p no_comment -m gcc-testresults@gcc.gnu.org |sh
> > tail: cannot open input
> >
> >         [...]
>
> The script moves the *.sum and *.log files to new names, so you can only
> run it once unless you move the files back to their original names.  You

Yes,. I re-ran the tests, then ran it once.
OK, I'll fiddle with that some more.

> shouldn't need to supply any options, since the default is to mail to
> the address you specified.  If you direct the output to a file then you

Good point!
> can look it over before running it, and that way it's available to
> modify and run again if there were problems the first time.
>
> Janis
>

        Thank you again,
        Hugh

Re: Meta-expectations when testing

[Janis Johnson]

On Wed, Feb 26, 2003 at 06:41:48PM +0000, Hugh Sasse Staff Elec Eng wrote:
> On Wed, 26 Feb 2003, Janis Johnson wrote:
> > The first sentence in test.html (generated from gcc/doc/install.texi) is:
> >
> >   Before you install GCC, we encourage you to run the testsuites and to
> >   compare your results with results from a similar configuration that have
> >   been submitted to the @uref{http://gcc.gnu.org/ml/gcc-testresults/,,
> >   gcc-testresults mailing list}.
> 
> Yes, but that is on the install page.

No, it's on the testing page, test.html, one of the 10 html files that
are generated from install.texi.

> > Would appending the following to that be sufficient?
> >
> >   Some of these archived results are linked from the build status lists
> >   at @uref{http://gcc.gnu.org/buildstat.html}, although not everyone who
> >   reports a successful build runs the testsuites and submits the results.
> 
> Not on the install page, for the reasons above.

Yes, but would it help at the beginning of test.html?
 
> > (Also, the maintainer of the build status lists sometimes doesn't have
> > time to go through the gcc-testresults archives after a release to
> > include new reports in the build status list as she ought to, and
> > welcomes mail pointing out the URLs of such reports, or patches to the
> > status list.)
> 
> There seems to be no address for a person on these pages, only for
> groups of people {gcc,gnu}@gnu.org, so I don't know who that is.

Sorry, that's me; I wouldn't imply that another volunteer is lazy.

Successful build reports to to gcc@gcc.gnu.org, as could informaiton
about the URLs for test results for a released compiler.  A patch to add
test result URSs to a build status list could go to
gcc-patches@gcc.gnu.org.

Janis

Re: Meta-expectations when testing

[Hugh Sasse Staff Elec Eng]

On Wed, 26 Feb 2003, Janis Johnson wrote:

> On Wed, Feb 26, 2003 at 06:41:48PM +0000, Hugh Sasse Staff Elec Eng wrote:
> > On Wed, 26 Feb 2003, Janis Johnson wrote:
> > > The first sentence in test.html (generated from gcc/doc/install.texi) is:
> > >
> > >   Before you install GCC, we encourage you to run the testsuites and to
> > >   compare your results with results from a similar configuration that have
> > >   been submitted to the @uref{http://gcc.gnu.org/ml/gcc-testresults/,,
> > >   gcc-testresults mailing list}.
> >
> > Yes, but that is on the install page.
>
> No, it's on the testing page, test.html, one of the 10 html files that
> are generated from install.texi.

*Sorry*, I misread that -- trying to read it too quickly!
>
> > > Would appending the following to that be sufficient?
> > >
> > >   Some of these archived results are linked from the build status lists
> > >   at @uref{http://gcc.gnu.org/buildstat.html}, although not everyone who
> > >   reports a successful build runs the testsuites and submits the results.
> >
> > Not on the install page, for the reasons above.
>
> Yes, but would it help at the beginning of test.html?

Yes, it would. Further apologies for my braindead response. Thank
you for your patience.
>
> > > (Also, the maintainer of the build status lists sometimes doesn't have
> > > time to go through the gcc-testresults archives after a release to
        [...]
> > There seems to be no address for a person on these pages, only for
> > groups of people {gcc,gnu}@gnu.org, so I don't know who that is.
>
> Sorry, that's me; I wouldn't imply that another volunteer is lazy.

Absolutely not: I read that as being *busy*.  Thank you.
>
> Successful build reports to to gcc@gcc.gnu.org, as could informaiton
> about the URLs for test results for a released compiler.  A patch to add
> test result URSs to a build status list could go to
> gcc-patches@gcc.gnu.org.

OK.
>
> Janis
>
        Thank you again,
        Hugh