From: Hans-Peter Nilsson <hp@bitrange.com>
To: Geoff Keating <geoffk@geoffk.org>
Cc: GCC Patches <gcc-patches@gcc.gnu.org>
Subject: Re: btest-gcc.sh patch ping
Date: Tue, 04 Dec 2007 22:26:00 -0000 [thread overview]
Message-ID: <20071204162649.D81194@dair.pair.com> (raw)
In-Reply-To: <A31E7556-E552-4698-8B4C-CEEF214C890B@geoffk.org>
On Mon, 3 Dec 2007, Geoff Keating wrote:
> Part of the integrity of the result is that the same things are tested in
> every run of the script on a particular platform. The script was specifically
> intended to not be configurable in the manner you want, because that would
> allow for the possibility that a misconfigured tester might miss some tests
> that ought to be run.
The same original set of tests are still run on all platforms.
If there are more that "ought to be run", I suggest you add
them. Regarding missing tests, you must mean some other patch.
The way $BTEST_GCC_EXTRA_TESTLOGS is added in the patch, just as
with the "static" logs, it will be an error if the file(s) do
not exist and contains "PASS:". ("More stable" than the
optional test logs added on an "if exists" basis above the
patched area, for example.)
> > > If there's a log missing, and you think the underlying functionality is
> > > stable
> > > and well supported by GCC contributors on all platforms where it is
> > > enabled,
> > > propose a patch to add it.
> >
> > The fortran tests are stable where it works, or so it seems.
> > Whether it works is port-specific (build issue on *-elf being
> > covered in this thread), and the patch above was a way to add
> > the functionality per-platform. It would also allow e.g.
> > tracking gas or ld results in a combined tree.
>
> The patch does not accomplish this. It instead adds the functionality
> *per-tester*. That's different to per-platform.
It does too, but better: rather than the static list you seem to
imply, it's enables the *tester* control over which optional
testlogs apply to which platform.
> > I just need a regtest-script I can point to, such that people can see how I
> > build and regtest and possibly repeat it.
>
> I would not go look at the scripts if I was trying to reproduce the results of
> my tester. I would instead look at the build log which shows the exact
> 'configure' and 'make' commands used, with no need to do shell variable
> substitution in my head.
By "repeat" I did not mean a specific run but to *reusing the
script for their own setup*. The build_log just as before
contains all details about the specific run, including whatever
value BTEST_GCC_EXTRA_TESTLOGS had at the time.
> > Your btest-gcc.sh mostly fits, but not covering all gcc-related tests that
> > run by "make check" is really an issue.
>
> The patch does not implement this, either, but it might be a good idea.
Yes it does too again: the tester will have to know beforehand
which logs apply, but I think that makes more sense than using
"find" and adding all testsuite logs willy-nilly.
> The reason the script doesn't cover all tests is because some parts of the
> testsuite were too unreliable to be used in an automated fashion. That was
> some time ago and it may be that today it should simply look at all the
> testsuite. Or, not. If you have some data that points either way it would be
> very helpful.
I wouldn't do it by default, but I agree e.g. a --find-all-logs
option might be useful. I don't have a use for it myself, I
just meant it as an example of what the patch enables.
> > I guess I could solve that by adding a variant of that script in contrib/
> > though I do prefer improving the existing one.
> Why?
>
> I can't imagine you're saving an enormous amount of time in maintenance. Nor
> is this script more conveniently located than a script you maintain in your
> own tree. You could put your script on a web server and point people at that.
I could say your arguments speak more against
contrib/regression/btest-gcc.sh being there, but I don't agree
with your arguments in either case.
On the contrary, I think there's a use for a script dealing with
the housekeeping of an automatic regression tester, and with
some optional aspects enabling different setups. I just mistook
your script for being it. I also think the place for such a
script is naturally in contrib/ - is really not the convenience
of having the script there, both for maintenance and sharing
obvious to you? If you mean something else, speak in clear text.
I propose adding a "less stringent" version as
contrib/build-and-regtest-gcc.sh.
If someone already has their own, different, script for
autotesting they are willing to contribute and is open to
changes while maintaining the default functionality, a script
which is no less than btest-gcc.sh (e.g. handles adding new
tests and pruning old ones), I don't mind switching my
autotester over.
brgds, H-P
next prev parent reply other threads:[~2007-12-04 22:26 UTC|newest]
Thread overview: 83+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <46EFBCC1.6070200@gmail.com>
[not found] ` <46EFC383.7020503@t-online.de>
[not found] ` <46EFC9E9.7090201@gmail.com>
[not found] ` <46EFCEF9.3060304@t-online.de>
[not found] ` <46EFCF7A.2080704@gmail.com>
[not found] ` <46EFD236.6080907@t-online.de>
[not found] ` <46EFDA4D.3070006@gmail.com>
2007-11-27 15:35 ` Link tests after GCC_NO_EXECUTABLES Bernd Schmidt
2007-11-27 22:17 ` Mark Mitchell
2007-11-27 22:40 ` Bernd Schmidt
2007-11-27 22:43 ` Mark Mitchell
2007-11-27 22:58 ` Bernd Schmidt
2007-11-27 23:17 ` Mark Mitchell
2007-11-28 0:23 ` Bernd Schmidt
2007-11-28 0:33 ` Mark Mitchell
2007-11-28 1:03 ` Bernd Schmidt
2007-11-28 1:24 ` Mark Mitchell
2007-11-28 6:34 ` Bernd Schmidt
2007-11-28 10:37 ` Mark Mitchell
2007-11-28 15:47 ` Bernd Schmidt
2007-11-28 8:16 ` Joseph S. Myers
2007-11-28 10:01 ` Mark Mitchell
2007-11-28 11:37 ` Joseph S. Myers
2007-11-28 11:40 ` Mark Mitchell
2007-11-28 15:37 ` Bernd Schmidt
2007-11-28 13:07 ` Richard Sandiford
2007-11-28 16:06 ` Rask Ingemann Lambertsen
2007-11-28 16:59 ` Daniel Jacobowitz
2007-11-28 18:55 ` Mark Mitchell
2007-11-28 22:41 ` Richard Sandiford
2007-11-28 23:03 ` Rask Ingemann Lambertsen
2007-11-29 6:21 ` Mark Mitchell
2007-11-30 21:07 ` Rask Ingemann Lambertsen
2007-11-30 21:08 ` Mark Mitchell
[not found] ` <20071130211005.GQ17368@sygehus.dk>
2007-12-01 9:48 ` Richard Sandiford
2007-12-01 11:53 ` Rask Ingemann Lambertsen
2007-12-01 12:03 ` Rask Ingemann Lambertsen
2007-12-01 13:37 ` Andreas Schwab
2007-12-01 22:35 ` Rask Ingemann Lambertsen
2007-12-02 21:11 ` Mark Mitchell
2007-12-05 17:22 ` Rask Ingemann Lambertsen
2007-12-06 0:38 ` Mark Mitchell
2007-12-06 17:58 ` Rask Ingemann Lambertsen
2007-12-07 1:37 ` Mark Mitchell
2007-12-07 15:31 ` Rask Ingemann Lambertsen
2007-12-07 18:39 ` Mark Mitchell
2007-12-07 21:48 ` Rask Ingemann Lambertsen
2007-12-07 21:57 ` Mark Mitchell
2007-12-13 22:25 ` [PATCH v2] " Rask Ingemann Lambertsen
2007-12-30 13:46 ` Ping " Rask Ingemann Lambertsen
2007-12-30 17:41 ` Paolo Bonzini
2007-12-07 11:27 ` Richard Sandiford
2007-12-07 15:18 ` Rask Ingemann Lambertsen
2007-12-08 11:25 ` Richard Sandiford
2007-12-04 14:46 ` Rask Ingemann Lambertsen
2007-12-05 0:10 ` Hans-Peter Nilsson
2007-12-05 0:19 ` Rask Ingemann Lambertsen
2007-12-05 0:46 ` Hans-Peter Nilsson
2007-11-30 3:56 ` Richard Sandiford
2007-11-30 5:32 ` Mark Mitchell
2007-11-30 8:07 ` Benjamin Kosnik
2007-11-30 9:58 ` Rask Ingemann Lambertsen
2007-11-30 11:41 ` Mark Mitchell
2007-11-30 23:25 ` Rask Ingemann Lambertsen
2007-11-30 13:16 ` Richard Sandiford
2007-11-30 20:32 ` Rask Ingemann Lambertsen
2007-11-30 21:10 ` Mark Mitchell
2007-11-30 21:15 ` DJ Delorie
2007-12-01 9:55 ` Richard Sandiford
2007-12-02 21:01 ` Mark Mitchell
2007-12-03 15:55 ` Richard Sandiford
2007-12-03 14:40 ` btest-gcc.sh patch ping and " Hans-Peter Nilsson
2007-12-03 16:07 ` Richard Sandiford
2007-12-03 22:43 ` Hans-Peter Nilsson
2007-12-03 16:59 ` Geoff Keating
2007-12-03 22:39 ` Hans-Peter Nilsson
2007-12-04 6:52 ` Geoff Keating
2007-12-04 22:26 ` Hans-Peter Nilsson [this message]
2009-03-28 14:10 ` Richard Guenther
2009-03-28 20:37 ` Geoff Keating
2007-11-30 20:49 ` Rask Ingemann Lambertsen
2008-05-15 21:40 ` Bernd Schmidt
2008-05-15 23:02 ` Richard Sandiford
2008-05-15 23:10 ` Bernd Schmidt
2008-05-16 0:18 ` Mark Mitchell
2008-05-16 2:21 ` Joseph S. Myers
2008-06-18 21:05 ` Ralf Wildenhues
2008-06-19 3:57 ` Mark Mitchell
2008-07-07 20:14 ` Ralf Wildenhues
2007-11-28 8:54 ` Joseph S. Myers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20071204162649.D81194@dair.pair.com \
--to=hp@bitrange.com \
--cc=gcc-patches@gcc.gnu.org \
--cc=geoffk@geoffk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).