Re: Patch for TearDownProcess

[Kris Van Hees <kris.van.hees@oracle.com>]

In view of the quote below, and history over the past couple of months, I'd
like to pose an important question to the overall developer community behind
Frysk:

	Should we strive to have a consistent, dependable testsuite?

The reason to ask is that on one hand, it seems some people prefer to look
at tests as individual units, regardless of their side-effects on later tests
in the testsuite (see the past discussions about the file descriptor leak
testing and its effects on the testsuite), whereas others prefer to see the
testsuite as a collection of tests that needs to be capable of producing
dependable results regardless of the outcome of one or more individual tests.

Right now it seems that not everyone is even on a specific side of the issue,
preferring one behaviour for some test cases, and the other for other test
cases.  The end result is that the Frysk testsuite is not producing results
that can be considered a valid indicator in terms of correctness.

In addition, just yesterday we encountered a case where some build-time testing
is conditional, causing a commit to go through (tested on F7 prior to commit)
that broke the build on other platforms (FC6) because on the latter platform
the test was enabled.  That kind of thing obviously gives developers a wrong
impression on whether their patch is going to be break builds or not.

Today I am seeing the following (first time I say this was July 3rd, and it
was reported in bugzilla):

| Running testDataFuncPeekBytes(frysk.sys.TestPtrace) ...PASS
| Running testDataFuncPeekBytes(frysk.sys.TestPtrace) ...ERROR
|   close: Bad file descriptor (fd 0)
| Running testLengthUnderBound(frysk.sys.TestPtrace) ...PASS
| Running testLengthUnderBound(frysk.sys.TestPtrace) ...ERROR
|   close: Bad file descriptor (fd 0)
| Running testOffsetUnderBound(frysk.sys.TestPtrace) ...PASS
| Running testOffsetUnderBound(frysk.sys.TestPtrace) ...ERROR
|   close: Bad file descriptor (fd 0)
| Running testLengthOverBound(frysk.sys.TestPtrace) ...PASS
| Running testLengthOverBound(frysk.sys.TestPtrace) ...ERROR
|   close: Bad file descriptor (fd 0)
| Running testOffsetOverBound(frysk.sys.TestPtrace) ...PASS
| Running testOffsetOverBound(frysk.sys.TestPtrace) ...PASS
| Running testLengthOnBound(frysk.sys.TestPtrace) ...ERROR
|   close: Bad file descriptor (fd 0)
| Running testOffsetOnBound(frysk.sys.TestPtrace) ...PASS
| Running testOffsetOnBound(frysk.sys.TestPtrace) ...PASS

While it can be argued that a test reporting its result twice is hardly an
issue, I would think that a single test reporting both a PASS *and* an ERROR
result would be more reason for concern...

In all, most development methodologies put a lot of value in testing.  It is
also my experience that even without a formal methodology, testing tends to
be considered rather important.  I do not believe that the Frysk testsuite can
be taken seriously as long as we cannot have a consistent policy in terms of
test quality.  Either test we allow failing tests to potentially corrupt the
validity of following tests, or we prohibit failing tests from corrupting
the validity of following tests.  Having it one way in some cases, and the
other way in some other cases simply leads to not being able to trust the
results of the testsuite at all.

	Cheers,
	Kris

On Fri, Sep 07, 2007 at 09:30:28AM -0400, Andrew Cagney wrote:
> Given the choices between a potential test-suite hang, and tear-down 
> leaving waitpid events around, the decision was made in favor of the 
> latter.  That decision hasn't changed.  Having the test run hang, is a 
> lesser evil then the test-suite continuing but producing bogus results.. 
> I restored the old behavior; and then added a timeout.  It currently logs a 
> message, I suspect it should abandon the test run, since the problem state 
> hasn't gone away.
>
> Andrew
>