Re: improving testing scheme (was Re: ACATS and GCC testsuite)

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Richard Kenner]

    If this is really impossible for you to do right, make a little script
    that does the necessary changes. You are making very heavy weather of
    a simple matter.

But remember that those "scripts" will have to be site-dependent, so their
creation will add yet more complexity to the configuration process.  The
issue isn't what "possible" or "impossible", but which of the two 
procedures is the most *reliable*.

The bottom line to me is that I don't see the major advantage in changing to
testing an installed compiler over what we do now.

Right now, "make check" is run from the build directory and tests the
compiler that was just built into that directory.  This is fairly
"idiot-proof": if the compiler wasn't built in that directory, it won't work
and there is no other place that "make check" looks for the compiler.  So if
somebody says they've run "make check", we know they've done it on the
compiler they in fact made their change on.

The single disadvantage of doing it this way is that we don't test "make
install".  However, I don't see this as a major problem: we don't change it
very often.  Moreover, to verify you've properly installed the compiler, you
don't need to run thousands of C programs through it: a handful are fine.  So
I think the testing of "make install" is of a different type, more like
sanity checking.

The risk, in my opinion, of changing "make check" to use the compiler in
PATH, rather than in the build directory, is that a lot of people are going
to accidentally run the check on the wrong compiler.  The main reason for
this will simply be "inertia": they are use to "make check" doing something
else for quite a while.  So it's easy to forget either "make install" or
changing the path, or both.  Having the maintainer-mode installation
prompt the user for configuration information and make new rules to
accomodate this is possible, but adds more complexity.

I think the risk of having "make check" run on the wrong compiler is far
larger than the risk of problems in "make install" that don't get detected.

I do think it worthwhile to add a new target "make check-install" that does a
(much shorter) check on the compiler in PATH to make sure everything is
installed.

I've also heard a proposal to rearrange things so the compiler is built
directly into a directory structure identical to that created by "make
install", so the latter is simply a bulk copy.  This will also solve the
problem without creating new ones.

Can you explain more about why you think the risk of procedural error here is
small enough that the small gain of testing "make install" overrides it?

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Paul Jarc]

kenner@vlsi1.ultra.nyu.edu (Richard Kenner) wrote:
> The risk, in my opinion, of changing "make check" to use the compiler in
> PATH, rather than in the build directory, is that a lot of people are going
> to accidentally run the check on the wrong compiler.

"make check" could look at the configuration in the build directory to
find out where the compiler should have been installed, and look there
rather than in $PATH.


paul

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Paul Jarc]

Laurent Guerby <guerby@acm.org> wrote:
> On Sat, 2003-03-29 at 00:25, Nathanael Nerode wrote:
>> A lot of people mentioned the justification "You want to test it before
>> you install it, don't you?".
>
> Well I'm probably stupid, but it makes no sense to me, what's magic
> about installing?

It often replaces the old, known-good version.  But if you install to
a unique, initially-empty prefix, then it makes more sense to install
before testing.


paul

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Paul Jarc]

kenner@vlsi1.ultra.nyu.edu (Richard Kenner) wrote:
>     "make check" could look at the configuration in the build directory to
>     find out where the compiler should have been installed, and look there
>     rather than in $PATH.
>
> Sure, but the problem there is that it has no way to know if the install
> actually *was* done,

So create a way.  Modify "make isntall" to change something in the
build directory (create a flag file, etc.), and have "make check" look
for that change.


paul

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Richard Kenner]

    "make check" could look at the configuration in the build directory to
    find out where the compiler should have been installed, and look there
    rather than in $PATH.

Sure, but the problem there is that it has no way to know if the install
actually *was* done, leaving the risk of somebody doing a make, then a
make check, forgetting to do the make install, and thinking they tested
their changes when they actually didn't.

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Robert Dewar]

> Nothing involving manually changing things like PATH is "100% reliable".
> It's always too easy to forget to change PATH or forget to change it back.
> 
> A testing procedure that depends on such manual operations is inherently
> error-prone because of the significant risk of forgetting to do them.

If this is really impossible for you to do right, make a little script
that does the necessary changes. You are making very heavy weather of
a simple matter.

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Richard Kenner]

    You install somewhere, put in PATH, play if it, if it doesn't please
    you, you change your PATH, if you're short on space you delete the
    install location. 100% reliable, safe 

Nothing involving manually changing things like PATH is "100% reliable".
It's always too easy to forget to change PATH or forget to change it back.

A testing procedure that depends on such manual operations is inherently
error-prone because of the significant risk of forgetting to do them.

Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[Laurent Guerby]

On Sat, 2003-03-29 at 00:25, Nathanael Nerode wrote:
> Laurent Guerby said:
> >That's why I by far prefer to have the testing being done
> >after install, and not in build like it is done for GCC
> >via DejaGNU now. It looks like I have no chance
> >of changing this sad state of affair, but well I
> >can still make a separate package for Ada.
> 
> A lot of people mentioned the justification "You want to test it before 
> you install it, don't you?".

Well I'm probably stupid, but it makes no sense to me, what's magic
about installing? You install somewhere, put in PATH, play if it,
if it doesn't please you, you change your PATH, if you're short on space
you delete the install location. 100% reliable, safe and feel good
transaction semantics, and you're exactly in user situation with
no fragile hypothesis about release time.

> However, we would certainly like to make the in-build testing situation 
> a lot closer to the post-install testing situation by eliminating the 
> many curious flags which currently need to be passed to the tested 
> programs.  This is one of the motivations behind Per Bothner's "new 
> build scheme", which I'm still working on, albeit slowly.

Looks like overly complicated to my small brain, which
has just given up on the idea of doing anything with
the current in build Makefile mess and DejaGNU.

-- 
Laurent Guerby <guerby@acm.org>

improving testing scheme (was Re: ACATS and GCC testsuite)

[Nathanael Nerode]

Laurent Guerby said:
>That's why I by far prefer to have the testing being done
>after install, and not in build like it is done for GCC
>via DejaGNU now. It looks like I have no chance
>of changing this sad state of affair, but well I
>can still make a separate package for Ada.

A lot of people mentioned the justification "You want to test it before 
you install it, don't you?".

However, we would certainly like to make the in-build testing situation 
a lot closer to the post-install testing situation by eliminating the 
many curious flags which currently need to be passed to the tested 
programs.  This is one of the motivations behind Per Bothner's "new 
build scheme", which I'm still working on, albeit slowly.

--Nathanael