Re: Testsuite documentation

Re: Testsuite documentation

[Mike Stump]

> Date: Sun, 26 Nov 2000 23:03:52 +0100 (CET)
> From: Gerald Pfeifer <pfeifer@dbai.tuwien.ac.at>
> To: <gcc@gcc.gnu.org>

> While working on that today, I noticed that the documentation seems to
> be incomplete.

> I'd really appreciate if those of you familiar with our testsuites could
> have a look at http://gcc.gnu.org/install/test.html and provide patches
> or feedback (and I'll write up the documentation then).

> Mike? ;-)

Well, I am the wrong person.  See, I know exactly how it works, and
how to make it sing already.  The ideal person, would be someone that
knows how to build and insall GNU software, but has never tested gcc
before.  If they can follow it and understand it...

I glanced at it, and what is there seems to be about right.

We could make it slightly easier to use and submit results by
including the build and submission of results into a make rule.  We
then document just the make rule.

Re: Testsuite documentation

[Gerald Pfeifer]

On Sun, 26 Nov 2000 (sic!), Mike Stump wrote:
>> I'd really appreciate if those of you familiar with our testsuites could
>> have a look at http://gcc.gnu.org/install/test.html and provide patches
>> or feedback (and I'll write up the documentation then).
> Well, I am the wrong person.  See, I know exactly how it works, and
> how to make it sing already.  The ideal person, would be someone that
> knows how to build and insall GNU software, but has never tested gcc
> before.  If they can follow it and understand it...

Okay, I think we've got a volunteer. His e-mail address is
<pfeifer@dbai.tuwien.ac.at>, and he'll contact you shortly,
should he have any problems. ;-)

> I glanced at it, and what is there seems to be about right.

I believe I found a missing item: Unlike PASS, FAIL, and XFAIL, we
don't describe XPASS.

How about:

  XPASS: the test unexpectedly passed

> We could make it slightly easier to use and submit results by
> including the build and submission of results into a make rule.
> We then document just the make rule.

Sounds like a good idea to me!

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

PATCH Re: Testsuite documentation

[Gerald Pfeifer]

On Wed, 17 Jan 2001, Gerald Pfeifer wrote:
>> I glanced at it, and what is there seems to be about right.
> I believe I found a missing item: Unlike PASS, FAIL, and XFAIL, we
> don't describe XPASS.
>
> How about:
>
>   XPASS: the test unexpectedly passed

I didn't get any response to this, but installed the following patch
nevertheless.

Gerald

Index: test.html
===================================================================
RCS file: /cvs/gcc/wwwdocs/htdocs/install/test.html,v
retrieving revision 1.29
diff -u -3 -p -r1.29 test.html
--- test.html	2001/03/08 22:45:26	1.29
+++ test.html	2001/03/11 08:57:23
@@ -94,12 +94,13 @@ results, the *.sum files summarize the r
 all the tests that have been run with a corresponding status code:</p>

 <ul>
-<li>PASS: the test passed as expected
-<li>FAIL: the test unexpectedly failed
-<li>XFAIL: the test failed as expected
-<li>UNSUPPORTED: the test is not supported on this platform
-<li>ERROR: the testsuite detected an error
-<li>WARNING: the testsuite detected a possible problem
+<li>PASS: the test passed as expected</li>
+<li>XPASS: the test unexpectedly passed</li>
+<li>FAIL: the test unexpectedly failed</li>
+<li>XFAIL: the test failed as expected</li>
+<li>UNSUPPORTED: the test is not supported on this platform</li>
+<li>ERROR: the testsuite detected an error</li>
+<li>WARNING: the testsuite detected a possible problem</li>
 </ul>

 <p> It is normal for some tests to report

Re: Testsuite documentation

[Tim Prince]

The test suites are there for a number of people who aren't in a position to
be patching the compiler; they may well be totally unfamiliar with compiler
internals but simply wishing to assure that they have built a gcc which is
up to snuff.

I've been confused myself on the --enable-languages and where it is meant to
work.  Even if that is used, the LANGUAGES option is useful to avoid
building those components which cause bootstrap to fail; for example, my
primary requirement is to build g77.  If g77 is unable to build as part of
the bootstrap, I don't want to leave it out of the enabled languages, as I
will build it immediately after proceeding as far as possible with a g++
bootstrap, even if I am building for a target where that also fails.  Should
people be required to find out by trial and error that various components of
gcc, such as chill, gcov, protoize, and the like, are not supported on their
target and are not necessarily required to test the correctness of their
build?

java is a good example.  It's not included in testsuite.  There is no need
to build it before running the testsuite.  If that changes, the
documentation should change accordingly.  As a recent post on the cygwin
list pointed out, it's near impossible for the uninitiated to know where to
get the library components needed to run gcj, and so gcj may become more
interesting when it does include a library and a testsuite.  Yes, I had a
working gcj on both cygwin and linux, and I expect to use java on the job
sooner or later, but why make it a stumbling block?

The environment variables which we have been required to set for testsuite
on cygwin, when the gcc builds were working on that target, are:

TCL_LIBRARY=/usr/share/tcl8.0
DEJAGNULIBS=/usr/share/dejagnu

I don't know whether there are other targets for which the necessary
environment settings don't just happen.

----- Original Message -----
From: "Gerald Pfeifer" <pfeifer@dbai.tuwien.ac.at>
To: "Tim Prince" <tprince@computer.org>
Cc: <gcc@gcc.gnu.org>
Sent: Monday, November 27, 2000 6:11 PM
Subject: Re: Testsuite documentation


> On Sun, 26 Nov 2000, Tim Prince wrote:
> > I'd suggest mentioning that LANGUAGES='c++ f77 objc gcov' is sufficient
to
> > build all the components which are tested in the standard test suite
>
> Hmm, what's the intention? For, if someone has to verify a patch to
> the backend, she has to bootstrap the entire compiler suite anyway,
> as part of the commit criteria. (And the list above just omits java,
> right?)
>
> Furthermore, isn't LANGUAGES deprecated in favor of --enable-languages
> now?
>
> > Maybe some explanation of what is to be expected of the mangling tests
> > (not clear on that myself), or the environment variable settings
> > expected.
>
> Yes, certainly!
>
> Could someone contribute a bit of documentation on the mangling?
>
> And/or any relevant information on the environment settings that is
> missing from the current page?
>
> Gerald
> --
> Gerald "Jerry" pfeifer@dbai.tuwien.ac.at
http://www.dbai.tuwien.ac.at/~pfeifer/
>

Re: Testsuite documentation

[Gerald Pfeifer]

On Sun, 26 Nov 2000, Tim Prince wrote:
> I'd suggest mentioning that LANGUAGES='c++ f77 objc gcov' is sufficient to
> build all the components which are tested in the standard test suite

Hmm, what's the intention? For, if someone has to verify a patch to
the backend, she has to bootstrap the entire compiler suite anyway,
as part of the commit criteria. (And the list above just omits java,
right?)

Furthermore, isn't LANGUAGES deprecated in favor of --enable-languages
now?

> Maybe some explanation of what is to be expected of the mangling tests
> (not clear on that myself), or the environment variable settings
> expected.

Yes, certainly!

Could someone contribute a bit of documentation on the mangling?

And/or any relevant information on the environment settings that is
missing from the current page?

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

Re: Testsuite documentation

[Toon Moene]

Michael Meissner wrote:

> On Sun, Nov 26, 2000 at 11:03:52PM +0100, Gerald Pfeifer wrote:

> > Mike? ;-)

> Ummm, can we use both first and last names when talking to a specific
> individual, since 'Mike' is not exact enough (and in my case, even last names
> is not enough, since Marcus Meissner has also contributed :-)

Tsk, tsk - I never understood what's so hard about picking a unique
first and last name - and not going beyond the 6 character limit.

-- 
Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290
Saturnushof 14, 3738 XG  Maartensdijk, The Netherlands
GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html
GNU Fortran 95: http://g95.sourceforge.net/ (under construction)

Re: Testsuite documentation

[Michael Meissner]

On Sun, Nov 26, 2000 at 11:03:52PM +0100, Gerald Pfeifer wrote:
> While working on that today, I noticed that the documentation seems to
> be incomplete.
> 
> I'd really appreciate if those of you familiar with our testsuites could
> have a look at http://gcc.gnu.org/install/test.html and provide patches
> or feedback (and I'll write up the documentation then).
> 
> Mike? ;-)

Ummm, can we use both first and last names when talking to a specific
individual, since 'Mike' is not exact enough (and in my case, even last names
is not enough, since Marcus Meissner has also contributed :-)

-- 
Michael Meissner, Red Hat, Inc.
PMB 198, 174 Littleton Road #3, Westford, Massachusetts 01886, USA
Work:	  meissner@redhat.com		phone: +1 978-486-9304
Non-work: meissner@spectacle-pond.org	fax:   +1 978-692-4482

Re: Testsuite documentation

[Tim Prince]

I'd suggest mentioning that LANGUAGES='c++ f77 objc gcov' is sufficient to
build all the components which are tested in the standard test suite (gcov
not being supported for building on as many targets as the others).  Maybe
some explanation of what is to be expected of the mangling tests (not clear
on that myself), or the environment variable settings expected.

----- Original Message -----
From: "Gerald Pfeifer" <pfeifer@dbai.tuwien.ac.at>
To: <gcc@gcc.gnu.org>
Sent: Sunday, November 26, 2000 2:03 PM
Subject: Testsuite documentation


> While working on that today, I noticed that the documentation seems to
> be incomplete.
>
> I'd really appreciate if those of you familiar with our testsuites could
> have a look at http://gcc.gnu.org/install/test.html and provide patches
> or feedback (and I'll write up the documentation then).
>
> Mike? ;-)
>
> Gerald
>
> PS: I'm aware of at least on significant omission which should be easy to
> spot, so let's see whether I get feedback on that! :-) :-)
> --
> Gerald "Jerry" pfeifer@dbai.tuwien.ac.at
http://www.dbai.tuwien.ac.at/~pfeifer/
>

Testsuite documentation

[Gerald Pfeifer]

While working on that today, I noticed that the documentation seems to
be incomplete.

I'd really appreciate if those of you familiar with our testsuites could
have a look at http://gcc.gnu.org/install/test.html and provide patches
or feedback (and I'll write up the documentation then).

Mike? ;-)

Gerald

PS: I'm aware of at least on significant omission which should be easy to
spot, so let's see whether I get feedback on that! :-) :-)
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/